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Executive Summary 



The Office of Learning Technologies (OLT) was launched by Human Resources Canada 
(HRDC) in 1996 as part of an education and training strategy for adult learners. It was 
established as a contributions program. The original objectives of the OLT have been 
refocused to address economic changes and at the time of this evaluation the revised 
objectives of the OLT were: 

• To help increase the capacity of Canadians and their community to use learning 
technologies in order to build a knowledge-based economy and society; and 

• To help close the economic and social divide between those who have computer skills 
and those who do not. 

Three funding initiatives have been established under the OLT with specific funding 
parameters, objectives and target audiences: 

• The Learning Technologies Initiative (LTI) operated from 1996 to 1999. During those 
years, LTI objectives evolved but with a focus on testing the effectiveness of learning 
technologies in a variety of settings and with diverse groups of adult learners. 

• The Community Learning Networks Initiative (CLN) was established in 1998 to 
support time-limited (3-year) pilot projects in partnership with community organizations. 
These projects offered access to a variety of learning resources and assisted communities 
in establishing their own computer networks. 

• The Learning Technologies for the Workplace Program (LTW) was established 
in 2000 and partners with non-profit organizations, industry associations, and educational 
institutions to provide funds (on a cost-shared basis) to projects that demonstrate the 
direct application of a learning technology in the workplace. This funding initiative is 
specifically aimed at small and medium sized enterprises and at disadvantaged workers. 



Approach to the Evaluation 

Evaluating partnership style programs presents a number of practical challenges. In the 
case of the OLT, the program’s objectives are achieved indirectly through funding 
projects undertaken by project sponsors and partners. This means that program 
outcomes, benefits and impacts depend on organizations, groups and individuals 
that are beyond the direct influence of the program. As a result, in specific instances, 
final outcomes are not easy to track or measure. Also, program and project 
outputs/outcomes are highly diverse, end-user populations are highly diverse, and there 
are no common metrics (such as the number of job placements) on which performance 
may be quantified. 
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The evaluation methodology developed for the OLT evaluation recognized these 
challenges and attempted to address them in a number of ways. For example, the use of 
multiple lines of evidence was emphasized to allow for findings from one approach to be 
substantiated/corroborated by findings from other lines of evidence. Also, case studies 
were used to help illustrate and/or demonstrate program outputs and impacts. The case 
studies also added concreteness and provided an in-depth understanding of how impacts 
are actually achieved. The evaluation also included a survey of non-funded projects to 
provide a reference for the findings on funded projects. 

The main components of the evaluation approach included a review of the literature and 
pertinent documents, a review and analysis of administrative data, case studies, surveys of 
project sponsors, project partners and non-funded projects, and key informant interviews. 



Evaluation Findings 

Are the objectives of the OLT still valid and relevant? Information collected from the 
document review, case studies, key informant interviews and surveys of project sponsors 
and partners supports the need for a national program that encourages the use of learning 
technologies to help develop new technical skills and upgrade old skills, particularly in 
small, isolated communities. 

Both the document review and interviews with stakeholders indicated that learning 
technology and community learning networks provide an important vehicle for 
implementing widespread learning opportunities. At the same time, however, the general 
view was that a number of barriers must be overcome (e.g. a lack of awareness/acceptance 
of learning technologies; a lack of access to infrastructure, technology and funding). 
Stakeholders indicated that they see the OLT and the federal government as having an 
important role in facilitating infrastructure, developing and facilitating partnerships, 
and providing financial assistance and funding. 

How important is OLT project funding? Two-thirds (66%) of the projects that applied 
for but did not receive OLT funding did not proceed. According to non-funded 
applicants, the major reason that their project did not proceed was the lack of alternative 
sources of funding. 

Only a minority (6%) of the unaccepted projects proceeded in full without OLT funding. 
Another 28% proceeded in a reduced or substantially reduced form. Most of the projects 
that proceeded without OLT funding had a narrower scope of reach, mostly targeting the 
local community, and were less likely to have used/tested or developed a website, 
the Internet or specialized software. The majority of survey project sponsors (85%) 
indicated that they would have been unable to deliver the same project scope and 
activities without OLT funding. 

Did OLT contribute to the development and use of learning technologies in Canada in 
innovative ways? OTL funded projects have used, tested and developed a wide variety of 
learning technologies, including computer-based training, the Internet, websites and 
CD-ROMs, to respond to the needs of end-users. Projects adopted innovative 
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strategies built upon the input/expertise of partners to help tailor the learning 
technologies to their end-user groups. 

Evidence from the case studies, surveys and key informants suggests that the program has 
also provided access to workplace training through LTW funding and has supported 
projects aimed at skills upgrading and lifelong learning. 

Has OLT developed sustainable partnerships? The OLT has led to the development of 
partnerships with private sector, community and non-govemmental organizations. At least 
one new partnership was developed by 77% of the surveyed project sponsors, with each 
sponsor developing (on average) 3 new partnerships. The evidence also suggests that 
these partnerships will continue in some form after OLT funding support ends. 
Looking specifically at LTI funded projects, which generally ended by 1999, 73% of 
the partnerships were reported to have extended beyond the life of the project. 

Who was reached by the OLT? OLT was designed to reach adult learners, especially those 
who were members of designated equity groups (persons in mral/remote areas, women, 
youth, seniors, visible minorities, persons with disabilities and Aboriginal people). 

The evaluation encountered a number of difficulties in trying to identify end-users 
because the program never required the collection of such data from funded projects. 
The available evidence indicates that OLT projects had considerable success in 
reaching most of the designated equity groups. Three exceptions were young mothers. 
Aboriginal clients and ESP populations. 

Are projects created through OLT sustainable? The available evidence from the case 
studies, surveys and key informants suggests that many OLT projects would either 
continue in some form after OLT support ends, or produce outputs/products that will be 
used after funding ends. More than three-quarters (78%) of the surveyed project sponsors 
felt that their project would continue in some form after the expiration of OLT funding. 

Did OLT increase the sharing of knowledge about learning technologies? Project 
sponsors generally agreed that their interaction with the OLT has increased their 
knowledge of new approaches in learning technologies (88%), learning technologies 
information (80%), and trends in the use of learning technologies (59%). Lor project 
sponsors and partners, the primary advantage of partnerships was said to be the sharing of 
knowledge and skills to provide different perspectives and ideas on learning technologies 
use and development. 

Did OLT contribute to improved accessibility to learning technologies? Project sponsors 
and partners agreed that funded projects have broadened access to learning opportunities 
(95%) by helping develop the communities’ technological infrastructure (66%) and 
facilitating the development of community expertise in learning technologies (74%). 

All initiatives have improved accessibility to learning technologies by partnering with 
organizations with access to targeted end-users. The CLN initiative has had the strongest 
relationship to the community as evidenced through significant linkages to Community 
Access Program (CAP) sites. Almost half (44%) of all CLN projects were directly 
linked, through the project partner or sponsor, to a CAP site. Additionally, by funding 
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CLN projects facilitated by community colleges or institutes (22%) and non-profit 
organizations (38%) that serve the community, OLT has further increased accessibility to 
learning technologies. Although the CLN funded projects have a range of specific 
objectives, most were aimed at increasing learning skills (23%), reducing barriers (23%), 
or increasing accessibility to technologies (27%). 

The available evidence also indicates that OLT projects benefited end-users. End-user focus 
groups for three of the four CLN project case studies identified a number of impacts, 
including reduced social isolation, increased confidence in learning and a reduced aversion to 
using computers. Also, the survey of project sponsors indicated that OLT projects have 
helped workers take advantage of technology (82%), facilitated lifelong learning in the 
community (81%), and helped develop a more s ki lled workforce (77%). 

Is the OLT cost effective? The evaluation was unable to use standard methods in 
considering cost-effectiveness due to the lack of data. Therefore, the evaluation was 
limited to reporting on the perceptions of key informants in this area and incrementality 
in the leveraging of additional funds. 

The key informants generally saw the OLT management model as cost effective. In part, 
it was suggested that this cost-effectiveness comes from the ability to involve other 
organizations in OLT funded projects and to leverage a significant amount of resources 
by facilitating cooperation among organizations across the country. 



Areas for Improvement 

The broad over-reaching objective of the OLT (i.e. to reduce the economic and social 
divide between those who have computer skills and those who do not) needs to be recast 
in more realistic terms. The current objective of the OLT needs to be recast to be 
something that a program with a limited budget can realistically achieve. An example 
might be to make the objective to improve computer skill sets of end-users and reduce 
their reluctance/inability to use newer technologies. 

Program controls need to be put in place to minimize the risk that funds leveraged by 
OLT projects are not incremental, and to safeguard against overlap and duplication. 
Although there is evidence that OLT projects are leveraging additional funds, the current 
program design does not include safeguards to reduce the risk that leveraged funds are 
not incremental (i.e., would have been used for similar activities in the absence of the 
program). Similarly, given that some of the unaccepted projects gained funding from 
other sources, there is a need for the program to be able to demonstrate that it has 
safeguards in place to prevent, or at least limit, any overlap or duplication of activities 
funded by other sources (e.g. by coordinating activities with Non-governmental 
organizations (NGOs), other levels of government, etc.). 
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While OLT has been able to reach designated equity groups, certain groups may 
require additional program effort to be adequately reached. While OLT funded 
programs have been relatively successful in reaching persons in remote/rural areas, 
women and seniors, there has been less success in reaching the Aboriginal community, 
young mothers and ESL populations. Aboriginal groups, especially, may require a 
targeted initiative to address the multiple issues identified in serving this population. 

There is a need for better outcome tracking at the end-user level. There is a need to 
provide for the tracking of end-users, for example by requiring project participants 
to register with project staff. 

Project sponsors and partners lack experience partnering. Project sponsors often lack 
required knowledge to develop and mobili z e formal partnerships. Lack of knowledge of 
partnering impacts both the development and maintenance of partnerships and can lead to 
conflicts during project development. This is particularly problematic for non-profit 
organizations and community agencies. In general, organizations would like more 
direction and assistance from the OLT in terms of the development and maintenance of 
partnerships. 

OLT staffing levels and staff turnover have impacted the program. There is a perception 
that there is insufficient OLT staff to manage current projects. Further, high staff 
turnover within the OLT has created an impression that staff are not sufficiently familiar 
with the program or with learning technologies. Generally, project sponsors would like 
more interaction and direction from OLT staff. 

There is a strong perception that OLT should allocate more resources to understanding 
and disseminating best practices and lessons learned. It is felt that OLT can act as a 
data warehouse for learning technology information, research and findings from projects. 
Stakeholders also requested additional opportunities for project sponsor interaction and 
information sharing between and within project initiatives. 

Delays in receiving OLT funding (i.e.,from the developmental phase to pilot phase as a 
result of the application process) resulted in waning project momentum and interest 
among partners, sometimes resulting in partner drop out. Informants and sponsors 
suggested that there be a “fast track” procedure for obtaining funding between the 
developmental and pilot phases, for projects that are already running. The issue of 
waning partner interest was more salient when community organizations were involved, 
given the unstable nature of their internal funding and staff resources. 
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Management Response 



Management believes that the summative evaluation has identified the importance of the 
Office of Learning Technologies (OLT) in supporting the development and upgrading of 
learning and technical skills through technologies. The evaluation indicated that OLT 
contributes to the development and use of learning technologies by designated equity 
groups. It highlighted the singularity of the program in linking learning technologies with 
non-institutional learning, while maintaining an overall program focus of knowledge and 
skills enhancement for innovation. 

The evaluation also identified challenges facing the program. Among those challenges 
primarily was the need to recast the broad over-reaching objective of the program 
(i.e. to reduce the economic and social divide between those who have computer skills 
and those who do not) in keeping with the program’ s limited budget. 

Management agrees with the need to put in place program controls to minimize the risk that 
funds leveraged by OLT projects are not incremental, and to safeguard against overlap and 
duplication of funds. The program is standardizing the practice of requiring potential 
recipients to confirm other sources of proposed funding prior to approving a contribution. 
Applicants must provide a statement when completing funding application forms. 

It was observed that OLT should attempt to increase its reach of equity groups such as 
Aboriginal communities, young mothers and English as Second Language populations. 
This issue is being addressed in the current call for proposals which gives priority to projects 
focusing on Aboriginal groups. Subsequent calls for proposals will be constructed in a 
manner to give priority to designated equity groups based on an identified need. 

Management agrees with the concern expressed in the report about the need for better 
outcome tracking at the end-user level. The requirement for end-user data collection will 
be included in the new Results Based Management Accountability Framework. 
The program will work in partnership with Evaluation and Data Development (HRDC) 
and consult both with Treasury Board and stakeholders to ensure that an appropriate 
framework is put in place. 

More support from GET to sponsors in the area of partnership development was 
recommended. GET is planning to organize partnership development sessions for new 
sponsors. The sessions will cover the issue of partnership development and maintenance 
with a view to encouraging the establishment of mutually beneficial relationships in 
pursuing a common goal. 

A finding of the evaluation was that staff turnover and staffing levels have impacted the 
program. In November 2001, the program began a business improvement process to 
address a number of staff and organizational issues. As a result of that process, additional 
project officers have been hired and trained. Teaming plans and working tools were also 
created to facilitate their role. The number of files each project officer is responsible for 
has been reduced by 50-60 percent. Staff turnover is now within acceptable norms. 
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The dissemination of best practices and lessons learned will be expanded through 
regional forums and the publication of an annual report reviewing emerging practices. 
A policy discussion involving experts is also being considered. 

Delays in receiving OLT funding (especially in the transition between the developmental 
and the pilot phase) were identified as problematic for projects and can result in project 
slowdowns, loss of interest and potential withdrawal of partners. OLT has reviewed and 
streamlined the evaluation and approval process in place. 

Human Resources Development Canada (HRDC) management would like to thank those 
individuals who participated in the evaluation of the Office of Learning Technologies (OLT). 
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1. Introduction 



The Office of Learning Technologies (OLT) was launched by Human Resources Canada 
(HRDC) in 1996 as part of an education and training strategy for adult learners. It was 
established with an annual budget of $6 million. As a contributions program, the OLT 
encourages and supports initiatives of various public and private sector partners to 
expand opportunities for innovative learning through learning technologies. 

This report presents the results of a summative evaluation of the OLT. Previously, 
a formative evaluation^ had been conducted to review the design and implementation 
of the OLT and to provide information on the program’s progress towards achieving its 
objectives. The purpose of the summative evaluation is to assess whether there is evidence 
that the objectives of the OLT have been met through projects funded by the program. 

This report on the summative evaluation of the OLT is presented in the following sections: 

• Chapter 2 presents an overview of the OLT program; 

• Chapter 3 highlights the evaluation methodology; 

• Chapter 4 examines the relevance of OLT objectives; 

• Chapter 5 examines program administration and delivery - including the issues of 
incrementality and overlap and duplication; 

• Chapter 6 examines the OLT’s contribution to the long-term development and/or 
adaptation of learning technologies; 

• Chapter 7 examines the OLT’s contribution to partnership development; 

• Chapter 8 looks at the end-users of the program; and 

• Chapter 9 examines the available evidence on program impacts relating to project 
sustainability, and the impact of the program on project sponsors and partners, 
end-users, and the community. 

• Chapter 10 summarizes the findings of the evaluation and provides conclusions. 



* Evaluation of the Office of Learning Technologies, Evaluation and Data Development, HRDC. 
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2 . Program Description 



The OLT was established in June 1996 as a contributions program with three key objectives: 

• To promote the effective use of learning technologies; 

• To support assessment, research and testing related to the use of learning 
technologies; and 

• To increase the availability and sharing of knowledge and quality information about 
learning technologies. 

Changes in the Canadian economy, however, suggested the need to refocus the OLT’s 

objectives to better address the needs of Canadians with barriers to the use of learning 

technologies. Accordingly, the revised objectives of the OLT are: 

• To help increase the capacity of Canadians and their community to use learning 
technologies in order to build a knowledge-based economy and society; and 

• To help close the economic and social divide between those who have computer skills 
and those who do not. 

To achieve the OLT’s objectives, three funding initiatives were established with specific 

funding parameters, objectives and target audiences: 

• The Learning Technologies Initiative (LTI) was initially known as the OLT 
Contribution Program and operated from 1996 to 1999. During those years, 
LTI objectives evolved but the primary goal of the initiative remained the support 
of two-year projects that contributed to an increased understanding and awareness 
of learning technologies through learning technology assessment and research. 
To this end, the initiative focused on testing the effectiveness of learning 
technologies in a variety of settings and with diverse groups of adult learners. 
In 2000, LTI was replaced by the New Practices in Learning Technologies. 

• The Community Learning Networks Initiative (CLN) was established in 1998 
to support time-limited (3-year) pilot projects in partnership with community 
organizations. These projects offered access to a variety of learning resources 
and assisted communities in establishing their own computer networks. 
CLN community-controlled and Internet-based computer networks were aimed at 
furthering social and economic development and encouraging lifelong learning. 
The CLN initiative has provided two types of funding: developmental phase, 
and pilot project funding. Developmental funding was used to conduct research 
activities, define the scope and direction of project activities and develop 
partnerships, methodologies and evaluation mechanisms for the project. Pilot phase 
activities on all projects were to be completed by March 31, 2002. 
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• The Learning Technologies for the Workplace Program (LTW) was established 
in 2000 and partners with non-profit organizations, industry associations, and educational 
institutions to provide funds (on a cost-shared basis) to projects that demonstrate the 
direct application of a learning technology in the workplace. This funding initiative is 
specifically aimed at small and medium-sized enterprises and at disadvantaged workers. 
To be funded through LTW, the project must be sponsored by an incorporated non-profit 
organization or an organization that is related to the workplace, and one partner must be 
actively participating in the project. 
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3. Evaluation Methodology 



The summative evaluation of the OLT focuses on assessing whether the program 
objectives have been met through projects funded by the LTI, CLN, and LTW initiatives. 
Specifically the evaluation^ was designed to determine the following: 

• If the objectives of the OLT are still valid and relevant, 

• How important OLT funding was to the establishment of the projects, and 

• The extent to which the OLT contributed to the development and use of learning 
technologies by reaching target groups, creating partnerships and developing 
sustainable projects. 

Evaluating partnership styled programs designed to achieve their objectives through 
groups and organizations beyond the direct influence of the program presents a number 
of practical challenges. In the case of the OLT, the program’s objectives are achieved 
indirectly through funding projects undertaken by project sponsors and partners. 
The projects and project outputs are highly diverse, and their final outcomes/impacts 
depend on additional persons, groups and organizations that are also beyond the direct 
influence of the program. This means, in specific instances, the impacts (e.g. on target 
groups, partnerships and communities) are not easy to track or measure. 

The evaluation methodology developed for the OLT evaluation recognized there 
challenges and attempted to address them, for example, by: 

• Emphasizing the use of multiple lines for all evaluation questions; 

• Employing both qualitative and quantitative data collection methods; and 

• Making full use of case studies and site visits to provide evidence on project 
targeting, outputs and the usefulness of project activities and outputs. 

The main components of the evaluation approach are presented below. 



3.1 Document Review 

A document review was conducted to provide a context for the summative evaluation. 
The document review examined the current state of learning technology theory and 
practice within and outside Canada. Additionally, the document review addressed key 
issues and lessons learned from domestic and international learning technology program 
development, impacts and outcomes of learning technology implementations, and trends 
in learning technology. 



^ The Evaluation Matrix is contained in Appendix A. 
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3.2 Administrative Data Review and Analysis 

Data review activities were undertaken with three data files, one for each of the three 
funding initiatives (LTI, LTW and CLN). The data review of these files included 
projects that were funded and projects not recommended for funding (i.e., non-funded 
projects). For the purposes of the evaluation, withdrawn projects were not included in 
the analysis. 



Table 3-1 

Number of Reviewed Projects Funded and Unfunded by Initiative 


Type of Project 


Pilot 


Developmental 


Total Number 
of Projects 


LTI 


N/A 


N/A 


344 


LTW 


12 


19 


31 


CLN 


69 


125 


194 


Total 


81 


144 


569 



Note: “withdrawn” are not included 



There was no missing information at the project level. Although there were only a 
limited number of variables available in each database when the evaluation started, more 
were added as the evaluation progressed. The lack of a full range of variables for each 
project (e.g. including users/participants) is not unique to the OLT and has been 
encountered in the evaluation of many other HRDC programs. 

The specific information available for the projects funded by each of the three funding 
initiatives included project title, location, project sponsor name and address, project start 
and completion date, project type (pilot or developmental) and status (active completed, 
withdrawn, non-funded). The amount of funding received by the project from the OLT 
was also available. 

Prior to analysis, data coding was completed for province, organization type 
(e.g., non-profit organization, university, college, etc.), initiative (CLN, LTI, LTW), 
and project status (active, completed, closed, and non-funded). All organizations were 
placed into only one category, regardless of whether there was reason to code them into 
two or more categories. These additional data elements provided a basis for examining 
the distribution of funding applications and funding levels. 



3.3 Case Studies 

Case studies were conducted on ten OLT funded projects. Three case studies were 
conducted with LTI funded projects, three with LTW funded projects, and four with CLN 
funded projects. The LTW is the most recently implemented initiative and, therefore, 
most of the LTW case-study projects were in the early stages of completion. This means 
that the case study evidence for these projects should be considered preliminary in nature. 
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The case studies consisted of a one or two day site visit with each of the ten projects. 
Locations for the site visits included Vancouver, Victoria and Lumby (B.C.), 
Edmonton (Alta.), Winnipeg (Man.), London, Ottawa and Toronto (Ont.), Montreal 
(Que.), and Halifax (N.S.). 

Site visits assessed a range of different aspects of the OLT program, such as partnerships 
developed in the course of OLT projects, deliverables/outputs of learning technology 
projects, sustainability, and lessons learned from the OLT project. During the site visit, 
the facility was toured and supplementary materials provided by the sponsors were 
reviewed. At that time, any hardware obtained for the project and any software developed 
or used for the project, such as CD-ROMs and websites, were observed. 

All of the case studies collected information on project targeting and the types of 
outputs/activities of the project. The case studies also collected information on project 
impacts and often included structured interviews with project partners. 

Each case study also attempted to obtain information directly from end-users through 
interviews/focus groups. It was difficult to complete this part of the case study because 
of a lack of tracking end-users at the project level. In the end, however, focus groups 
(each with nine participants) were arranged with end-users at three of the four CLN case 
study sites. 

• Vancouver Community Network: Project to provide Internet tools/access and training 
at community sites, and targeted on individuals with low income and literacy skills. 
The end-user focus group was conducted with seniors; 

• Monashee Learning and Training Centre: Project to establish a hub/centre of service 
in the remote community of Lumby to facilitate increased access to lifelong learning 
and to increase the presence of on-line businesses. The end-user focus group was 
conducted with persons who had taken courses at the Centre; and 

• Creative Retirement Manitoba: Project aims to provide seniors with necessary 

government and community information by on-line access, combined with courses 
and mentoring programs to improve seniors’ computer skills. The end-user focus 
group was conducted with seniors. 

An interview guide was developed to use with project sponsors and partners. A site-specific 
moderator’s guide was developed for each case study site in which a focus group 
was conducted. 

Appendix B provides a summary of each of the case studies including the project target 
population/client group, client needs/project objectives, an assessment of the extent to 
which client needs/project objectives were met, barriers/factors affecting the achievement 
of the objectives, and other issues. 
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3.4 Surveys of Project Sponsors, Project Partners and 
Non-funded Applicants 

Three mail surveys (completed by mail and telephone follow-up) were conducted: 

• A project sponsor survey; 

• A project partner survey; and 

• A non-funded applicant (non-funded projects) survey. 

The surveys were designed to collect information on project activities and project 
partnerships, to determine satisfaction with the application process and OLT 
information/services, and to assess impacts of each project and the OLT program overall. 
All instruments were pre-tested and available in both English and French. 

All LTI (124), CLN (124) and LTW (25) funded project sponsors were mailed a 
questionnaire. However, only a sample of the 782 non-funded LTI (717), CLN (59), 
and LTW (6) applicants were sent a survey for completion. The selected sample of 
non-funded applicants was comprised of a random sample of the LTI non-funded 
projects (245), and all non-funded CLN (59) and LTW (6) applicants. 

The sample for the project partner survey was obtained by asking funded project sponsors 
for the names and contact information of up to three of their project partners. 
Questionnaires completed by funded project sponsors provided the names of 173 project 
partners, all of which were mailed a project partner survey questionnaire. 

Table 3-2 provides a detailed breakdown of the sample of funded project sponsors, partners 
and non-funded applicants who completed the mailed questionnaire. The final sample of 
project sponsors had lower proportions of CLN and LTW developmental projects. The final 
sample of non-funded projects had higher proportions of non-funded CLN and LTI 
applicants. This was due, in part, to the length of time between the survey and applicant 
submission of funding applications to the LTI initiative, which operated between 1996 and 
1999. The final response rate based on the sample is also included in Table 3-2. The sample 
was obtained by excluding cases where the organization that applied for or received funding 
no longer exists, or the contact person and all individuals involved in the application process 
were no longer at the organization and could not be tracked. 
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Table 3-2 

Survey Target Groups 





Population 


Sample 


Number of 
Completions 


Response Rate 


Project Sponsors 


283 


248 


126 


51% 


LTI 


124 


110 


62 


56% 


CLN 


Developmental 


68 


61 


20 


33% 


Pilot 


56 


53 


30 


57% 


LTW 


Developmental 


14 


13 


5 


38% 


Pilot 


21 


11 


9 


82% 


Project Partners 


173 


147 


59 


40% 


LTI 


71 


53 


19 


36% 


CLN 


Developmental 


26 


25 


13 


52% 


Pilot 


52 


47 


14 


30% 


LTW 


Developmental 


N/A 


N/A 


N/A 


N/A 


Pilot 


24 


22 


13 


59% 


Non-Funded Projects 


782 


207 


53 


26% 


CLN 


59 


47 


17 


36% 


LTI 


717 


154 


34 


22% 


LTW 


6 


6 


2 


33% 


TOTAL 


1,238 


602 


238 


40% 



3.5 Key Informant Interviews 

To complement the above lines of evidence, structured interviews were conducted with a 
broad range of key informants, resulting in a total of 17 key informant interviews 
(Table 3-3). 



Table 3-3 

Informant Interview Respondents 


Site 


Completed Interviews 


Provincial Government/Public Sector Officials 


3 


HRDC Regional Staff 


2 


Community Organizations 


3 


Partners 


3 


Educational Institutions 


3 


External Advisory Network 


3 


Total 


17 



The interview guide was developed to address the informant’s involvement in and 
perspectives on the OLT initiatives and issues related to learning technologies. The guide 
was pre-tested with a community organization representative to ensure that the questions 
were clear, meaningful, and presented in a logical sequence. Introduction letters were 
sent to informants prior to the interviews. Interviews were conducted in-person and by 
telephone in the official language preferred by the interviewee. Thank-you letters were 
sent to all interviewees after the interview was complete. 
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3.6 Strengths and Limitations of the 
Evaluation Approach 

The evaluation approach developed for the OLT recognized the range of practical 

difficulties in evaluating this type of partnership program: 

• Program outcomes, benefits and impacts depend on organizations, groups and individuals 
that are beyond the direct influence of the program. As a result, in individual instances, 
final outcomes of program funding are not easy to track or measure. 

• Program and project outputs/outcomes are highly diverse, end-use populations are 
also diverse, and there are no major common metrics (such as the number of job 
placements) on which performance may be quantified. 

• Obtaining data on project outputs, outcomes and end-users can be difficult, the available 
data can be uneven across projects, and key data for evaluation purposes may simply not 
be collected by project sponsors. 

• Program expectations and objectives can be broad and over-reaching. 

A number of steps were taken to address these practical difficulties in evaluating the OLT: 

• The multiple lines of evidence approach was emphasized to allow for findings from 
one approach to be substantiated/corroborated by findings from other lines of 
evidence. In the OLT evaluation, this allows for greater reliance and confidence to be 
placed in findings, especially those from non-objective sources (e.g. project sponsor 
survey respondents); 

• Case studies that illustrate and/or demonstrate program outputs and impacts are an 
important line of evidence, especially in the case of partnership styled programs 
involving varied outputs and a diverse end-user population. The case studies also add 
concreteness and provide an in-depth understanding/insights of how the impacts are 
actually achieved. In the OLT evaluation, ten case studies were conducted, with three 
of them involving focus groups of end-users; and 

• The OLT evaluation included a survey of non-funded projects to add an important 
perspective against which to compare the funded projects. 

Although the evaluation approach developed for the OLT recognized and attempted to 

address the challenges in evaluating this type program, the following limitations should 

be noted: 

• Evidence from the survey of project sponsors can be viewed as non-objective 
opinions from a group with a vested interest in program continuation. As sponsors 
are recipients of program funds, the findings from the survey alone must be 
interpreted with caution. Greater reliance can be placed upon the findings, however, 
in instances where they are corroborated from other lines of evidence. 
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Data on the end-users of project outputs were unavailable as the program never 
required the collection of such data from the funded projects. End-user data will not 
be consistently collected until a new RMAF is in place that sets out the expectations 
and means for collecting the information. In order to partially compensate for the 
unavailability of end-user data, focus groups were conducted with the end-users for 
three of the four CLN case studies. 
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4. Relevance of OLT Objectives 



This section examines whether the OLT objectives are still valid and relevant to the 
current situation by looking at whether there continues to be a need for a national 
program like the OLT to encourage the use of learning technologies. 

Information collected from the document review, case studies, key informant interviews 
and surveys of project sponsors and partners supports the need for a national program 
that encourages the use of learning technologies to help develop new technical skills 
and upgrade old skill - particularly in smaller, isolated communities. The literature 
indicates that there continues to be a need for citizens to develop new technical skills and 
up-grade old skills to fully participate in the current economy. Also, technology is 
changing the way work is performed in broad segments of many traditional industries and 
creating new industries that require new skills. 

In some communities there is sufficient population and institutional density to facilitate 
innovation."^ In others, however, the majority of the industrial development has been 
within narrow parameters such as primary resource extraction or processing sectors. 
Often single industry communities are remote from other population centres and have not 
developed the capacity for innovation beyond primary industries.^ Also, the citizens 
from single industry communities may not have the necessary technical skills to make a 
transition to employment in other industries or communities.^ 

Citizens who do not possess the required technical skills increasingly contribute to the 
Digital Divide. The challenge is to facilitate the development of processes and 
structures to allow equal access to learning technology for disadvantaged communities 
and disadvantaged individuals.^ 

The surveys and case studies indicated that barriers contributing to the Digital Divide 
include a lack of awareness/acceptance of learning technologies and lack of 
infrastructure, technology and funding. Key informants also cited lack of interest or 
attitudinal barriers as one of the four main factors contributing to the Digital Divide. 
Other factors cited by key informants included lack of infrastructure, lack of information 
sharing and low levels of technical literacy. 



^ Literature Review: The Use of Learning Technologies in Canada, written for the Summative Evaluation of the 
Office of Learning Technologies Program, Michael Gurstein. 

Porter, Michael E. (1990) The Competitive Advantage of Nations. London: Macmillan. 

^ Literature Review: The Use of Learning Technologies in Canada, written for the Summative Evaluation of the 
Office of Learning Technologies Program, Michael Gurstein 
^ Cape Breton County Stakeholders Assembly on the Economy, (1998), Report on Deliberations: Suggested 
Eramework for a Cooperative Action Plan on the Acute Economic Challenges of Cape Breton County. 

’ The refers to the economic and social divide between those who have computer skills and those who do not. 

* Literature Review: The Use of Learning Technologies in Canada, written for the Summative Evaluation of the 
Office of Learning Technologies Program, Michael Gurstein 
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A major difficulty for smaller, isolated communities is distance from learning networks, 
creative initiatives and research. Also, the literature indicates that communities without 
the means to determine which information infrastructures and technologies are most 
worthwhile for them may waste a great deal of time, effort and money on less than 
productive activities and on technical dead-ends. 

The literature and stakeholders confirmed that learning technology and community 
learning networks provide an important vehicle for implementing widespread learning 
opportunities. The literature also indicates that these learning networks can link local 
economic development strategies with community learning^ to facilitate human capital 
development. 

Representatives at the case study sites, the surveyed stakeholders and key informants also 
see a wide a variety of opportunities for the use of learning technologies in the 
development of community learning, including the use of learning technologies for 
the following purposes: 

• Employment skills development and re-training; 

• Small business education and e-commerce; 

• Increased access to workplace training and health and safety training; 

• Rural access to life-long learning; 

• Teaching ESL and individuals with disabilities; and 

• Providing links to community education. 

The document review indicates that there are few Canadian programs funding 
OLT-related activities. The literature review indicated that the OET is one of only 
two programs in Canada that links learning technologies with non-institutional 
learning, such as community or lifelong learning, while maintaining an overall 
program focus of skills development and training for innovation. The other example 
of this type of programming is the TeleEducation website in New Brunswick, which 
provides 50,000 on-line courses. 

Although a variety of programs exist at the provincial level to support adult and community 
learning, those concerned with introducing technologies to facilitate learning are few in 
number and scope. Within Ontario, for example, funds available for community-based 
technology initiatives'"^ have primarily been aimed at linking community technology with 
local economic development, rather than with community or lifelong learning. Similarly, 



^ Gurstein op.cit. http://olt-bta.hrdc.gc.ca/CLN/whatsnew_e.html, Canada. 
http://www.worldbank.org/poveity/scapital/index.htm 

** Literature Review: The Use of Learning Technologies in Canada, written for the Summative Evaluation of the Office 
of Learning Technologies Program, Michael Gurstein. 
http://courses.telecampus.edu/subjects/index.cfm 

Literature Review: The Use of Learning Technologies in Canada, written for the Summative Evaluation of the Office 

of Learning Technologies Program, Michael Gurstein. 

http://www.est.gov.on.ca/english/fp/fund_aindex.html 
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in Quebec, the primary focus for both extended education and technology enabled education 
has been the Tele-universite de TUniversite du Quebec/^ However, the Quebec government 
has not made investments in enabling technologies for lifelong learning/^ 

Stakeholders feel that the OLT should continue to aim at closing the economic and 
social divide between those who have computer skills and those who do not. A recent 
study concluded that the Digital Divide in Canada has continued to close between 1996 
and 2000. This finding, however, does not fully address the issue that a significant divide 
still remains in the use and access to information technologies between high and low income 
groups, between young and old, between urban and rural populations, and between those 
with higher levels of education compared to those with less education. When high and low 
income groups are compared, the Digital Divide continues to grow for the lowest income 
groups, albeit at a slower rate than in 1996. 

Project sponsors, project partners and key informants considered the federal government 
to have an integral and diverse role in promoting learning and skills development using 
learning technologies. The role of the federal government is seen to include providing 
information, conducting needs assessment, and supporting research around the issue of 
the Digital Divide. With respect to specific program or project development, 
stakeholders considered the federal government to have a key role in: 

• Facilitating infrastructure development and access; 

• Developing and facilitating partnerships; and 

• Providing financial assistance and funding. 

Project partners (84%), project sponsors (86%) and non-funded project applicants (76%) 
strongly agreed that assisting in closing the economic and social divide between people 
who have computer skills and Internet access and those who do not is an appropriate 
objective for the OLT. 

Key informants also agreed that the OLT should continue to work at closing the 
Digital Divide, particularly because there is no other federal government initiative or 
activity to address this issue. Many of the key informants also noted that the Digital 
Divide should be addressed through collaboration and partnership between all levels 
of government, community organizations, the educational system and the private 
sector. OLT was seen to serve in the role of “catalyst” between HRDC, Industry 
Canada and the provincial governments - through providing information and 
enhancing capacity to mobilize engagement. 



http://www.teluq.uquebec.ca/webteluq/index.html 

Literature Review: The Use of Learning Technologies in Canada, written for the Summative Evaluation of the Office 
of Learning Technologies Program, Michael Gurstein 

The Digital Divide in Canada, George Sciadas, Statistics Canada, Catalogue no. 56F0009XIE, 2002. 
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Through continued support of the OLT, stakeholders also feel the federal government has 
an important role as part of the national skills engagement process in disseminating 
information on best practices and acting as a data warehouse for learning technologies 
(i.e. a type of information brokerage role). 
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5. Program Administration and Delivery 



This section examines the administration and delivery of the OLT by examining non- 
funded projects, the ratio of funded to non-funded projects for the three funding 
initiatives, awareness of the OLT, and the level of satisfaction with the application 
process and project administration. 



5.1 Non-funded Projects 

OLT funding appears to be an important factor in the development of projects aimed at 
using technology to promote learning, with only 6% of the non-funded projects 
proceeding in full without OLT funding. The survey of non-funded project applicants 
indicated that only 6% of the non-funded applicants pursued the full project as outlined in 
the funding application (as shown in Table 5-1). Another 28% pursued less than the full 
project, with most of these projects going ahead in a substantially reduced form (i.e. less 
than 25% of the project that had been outlined on the OLT application). The other 66% 
of non-funded projects did not proceed at all after being unaccepted for funding by the 
OLT program. 



Table 5-1 

Number of Projects that Proceeded Without OLT Funding 


Percentage that Proceeded with Project 


Non-Funded Project Sponsors 
( Vaiid N=53) 


Project did not proceed: 


66% 


Project proceeded as: 


34% 


25% or less 


(19%) 


26% to 50% 


(6%) 


51% to 75% 


(2%) 


76% to 99% 


(1%) 


100% 


(6%) 


Total 


100% 



Applicants are applying for OLT funding when other sources of funding are not 
readily available. The major reason given by non-funded applicants for projects not 
proceeding, entirely or in part, was lack of alternative funding sources. As further 
indication that other sources of funding were not readily available, most (73%) 
non-funded applicants would apply again for OLT funding. Only 5% of those who 
would not apply again to OLT indicated it was because they felt they could obtain 
funding elsewhere. 

Projects that proceeded in the absence of OLT funding obtained alternate funding through 
donations (22%), internal sources (22%), government (22%), and other sources (34%). 
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Almost two-thirds of the unsuccessful applicants either did not receive or could not 
recall the reason for the rejection of their application. The file review conducted for the 
formative evaluation of the OLT indicated that the main reasons a project would not be 
recommended were: 

• Results are unlikely to have a broad impact; 

• Proposal requires significant reworking; 

• Sufficient similar activities being supported by OLT; 

• Results would add little to the existing knowledge base on LT; 

• Project does not represent an innovative use of LT; 

• Major portion of project is in purchase of capital expenditures; 

• Activities focus primarily on connectivity and communications; 

• Activities focus primarily on service delivery or provision of training; or 

• Activities focus primarily on transfer of media (e.g. paper to CD). 

The survey of the non-funded project applicants indicated that it was common for 
unsuccessful applicants to indicate they did not receive or could not recall the reason for the 
rejection of their application (64%). Among the thirty-six percent (36%) that could recall 
the reason for application rejection, it was common for the project to have not met OLT 
funding criteria (17%), or to have insufficient partnerships developed (1 1%). 

Looking specifically at the sub-group of non-funded applicants that proceeded with the 
project in the absence of OLT funding (which accounted for 34% of the non-funded 
projects), most (72%) could not recall or did not provide a reason for why their 
application was rejected. Only 6% of the non-funded applicants indicated their project 
did not fall within the scope of OLT objectives. 

For evaluation purposes, the non-funded “active” projects can serve in some respects as 
an adequate reference against funded projects. The suitability of using the non-funded 
“active” projects as a reference group was assessed by examining the main objectives of 
the proposed project as stipulated in the proposal application. The objectives of these 
projects varied greatly; however, most aimed at providing training or addressing the 
learning needs of equity groups. Therefore, non-OLT funded projects can serve in some 
respects as an adequate reference against the funded projects. Although these 18 projects 
do not provide an ideal comparison or control group, particularly given the relatively few 
projects that provided data, comparisons are made between the OLT funded projects and 
the unaccepted projects that proceeded in the absence of OLT funding where appropriate 
in the remaining sections of the report. 
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5.2 Projects Funded by Type of Initiative 

Among the three funding initiatives, LTW has the highest ratio of accepted applicants. 
Table 5-2 shows the number of applications submitted to each of the three OLT funding 
initiatives and the percentage of applications that were funded or failed to receive 
funding from the initiative. Information from the file review indicates that the LTW 
funded the highest proportion of applicants. However, it should be noted that this 
funding initiative had also run for the shortest duration at the time of this evaluation. 



Table 5-2 

Funded Projects as a Ratio of Total Applicants by Initiative 


Initiative 


Funded 


Non-funded 


Total 


LTI (N=344) 


36% 


64% 


100% 


CLN (N=194) 


69% 


31% 


100% 


LTW (N=31) 


81% 


19% 


100% 



5.3 Awareness of the OLT Initiative 

Project sponsors most often became aware of the OLT program through the OLT website 
(40%), and project partners usually became aware of the program through the sponsor’s 
funding application (61%). Project sponsors, commonly became aware of the OLT 
program through the OLT website (40%), national or provincial associations (21%), 
and OLT staff (15%). One-third (31%) of LTI funded project sponsors were notified by 
OLT, through email, of the call for proposals. Project partners primarily became aware 
of the program through the sponsor’s funding application (61%). 



5.4 Satisfaction with the Application Process and 
Project Administration 

Project sponsors, and those partners involved in the application process, were generally 
satisfied with the application process, although there was some confusion around the 
“matching funds” component of the application. Project sponsors and partners 
generally agreed on the following: 

• Program- initiative objectives were clear (82%); 

• Eligibility criteria set forth in the application guidelines were clear (77%); 

• Proposal assessment process was adequately described (62%); and 

• Selection criteria were clear (62%). 

At the same time, the case study interviews and key informant interviews suggested that 
there is an opportunity to improve the application process. For example, the application 
process was viewed as cumbersome or unclear in some respects, and in need of 
streamlining. In particular, applicants noted that they were unsure what constituted a 
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matching fund and felt that contributions in-kind should be permissible as matching funds 
by the OLT. For some, the inability to utilize contributions in-kind as matching funds had 
resulted in difficulties obtaining project partners. 

In the area of project administration, project sponsors and other stakeholders were 
most concerned with the time and effort required from project staff to meet the 
program reporting requirements. Resource allocation requirements for management of 
the OLT contribution arose as a significant project administration issue among project 
sponsors and partners. Organizations found that they generally utilized more staff 
resources to run the project than they had originally anticipated. In some cases, projects 
required hiring additional staff to meet mandatory OLT reporting requirements. Both key 
informants and project sponsors commented that the pressure placed on staff resources by 
the project would dissuade the organization from applying again for OLT funding. 

Reporting documents received from project sponsors were not optimally utilized by the 
OLT. There were significant limitations to the administrative data provided by the OLT: 
very few variables were captured in the database and no project outcome data was 
available. While case study interviews indicated that project sponsors had provided 
significant amounts of information through on-going reporting, little of this information 
was captured by the administrative data. Further, project sponsors expressed concern 
that, although OLT was provided with on-going progress reports, they never received 
feedback on project performance or feedback on project issues raised in the mandatory 
reports. Project sponsors and partners generally agreed (72%), however, that they were 
satisfied that OLT staff were prompt in addressing direct questions put forth by means 
other than mandatory reporting. 

Project sponsors and partners indicated that they would have liked more contact with the 
OLT during project development and roll out. The general view was that project 
sponsors and partners would have preferred to have more contact with program staff 
during the early stages of the project. Further, it was noted that there had been a lack of 
continuity due to staff turnover within OLT. 

Some projects experienced difficulties in developing and maintaining partnerships. Project 
sponsors and key informants noted that some difficulties were experienced in developing and 
maintaining partnerships. Partnership activities required considerable resources and were 
particularly challenging for those organizations inexperienced in partnering, (i.e. non-profit 
and community organizations). Larger organizations or educational institutions noted that 
community sponsors had little experience in partnering and therefore lacked understanding of 
what roles and responsibilities each partner should fulfill. 

The case studies indicated that organizations with little experience in partnership 
development and maintenance would have liked additional support from the OLT in 
partnering. In some cases partners indicated that the guidelines provided by OLT were 
difficult to follow and required interpretation and clarification by OLT staff. They noted that 
this created an additional barrier for smaller agencies and less experienced applicants. 
Similarly, the Vancouver Community Network (VCN), which partnered exclusively with 
non-profit and community agencies, found capacity building was a significant issue in project 
development. VCN’s partners were often unclear on the amount of internal resources 
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available to commit to the project, and were unable to focus sufficient resources on the 
project because of other organizational priorities. Key informants echoed the problems 
identified by project sponsors and the case studies. In addition, key informants indicated that 
a lack of understanding of OLT goals and objectives was also a barrier to partnership. 



5.4 . 1 Use of OL T Services 

Project sponsors found the OLT website and staff to be useful sources of support. 
Project sponsors commonly accessed the OLT website (84%) and OLT staff (87%) were 
considered to be useful sources of support. Additional services were accessed to a lesser 
degree, in part because they were newly offered or offered on a time-limited basis by the 
OLT. These services included the following: 

• OLT published reports/studies (52%); 

• The annual meeting of the Advisory Network of Experts and other forums (36%); and 

• OLT workshops (34%). 

At the same time, the case study interviews with project sponsors and partners indicated 
that some organizations had not accessed or were unfamiliar with many of the resources 
available through OLT. This was evidenced by requests for support documents currently 
in place at OLT such as partnering guidelines. 



5.4.2 Views on Cost Effectiveness 

The evaluation was unable to use standard methods in considering the cost-effectiveness 
of the OLT, because the data needed to undertake a standard analysis were not available. 
Therefore the evaluation was limited to reporting on the perceptions of key informants in 
this area and the related issue of incrementality in the leveraging of additional funds. 

The key informants generally saw the OLT management model as cost effective. 
When key informants were ask whether the OLT is cost efficient and effective in 
promoting learning technologies, they indicated that they saw the OLT management 
model as cost effective, particularly when compared to other programs. In part, it was 
suggested that this cost-effectiveness comes from the ability to involve other 
organizations in OLT funded projects and to leverage a significant amount of 
resources by facilitating cooperation among organizations across the country. 

Although there is some evidence that OLT projects leveraged additional funds, it is 
difficult to determine the extent to which the leveraged funds are incremental. The OLT 
program requires that OLT funding is matched one-for-one by contributions from project 
partners. An examination of the case study projects shows that significant resources, 
either in-kind or financial, were leveraged through the OLT funded projects. Information 
from 6 case studies (Table 5-3) showed that the actual leveraging was in the order of 
$1.50 for every $1 of OLT funding. 
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Table 5-3 

Leverage of Resources by OLT Funded Projects Visited in the Case Studies 



Case Study Site 
(Project) 


Dollars 

Leveraged 


Source of 
Contribution 


Amount of 
Contribution 


Type of 
Contribution 


U avenue, Centre 
Internet 

Communautaire 
Montreal, PQ 


$1/$1.17 


OLT Contribution 


$300,000.00 


Financiai 


Sponsor/Partner 

Contribution 


$350,000.00 


Financiai 


Totai 


$650,000.00 




Monashee Learning 
& Training Centre 
Lumby, BC 


$1/$1.90 


OLT Contribution 


$260,000.00 


Financiai 


Sponsor/Partner 

Contribution 


$493,519.00 


Financiai/ 

In-kind 


Total 


$753,519.00 




Vancouver 
Community Learning 
Network 
Vancouver, BC 


$1/$1.24 


OLT Contribution 


$197,500.00 


Financiai 


Sponsor/Partner 

Contribution 


$244,200.00 


Financiai/ 

In-kind 


Total 


$441,700.00 




Teaching Workplace 
Literacy as a Safety 
Initiative in the Forest 
Industry 

AB Forest Products 
Assoc. 

Edmonton, AB 


$1/$3.67 


OLT Contribution 


$150,000.00 


Financiai 


Sponsor/Partner 

Contribution 


$550,000.00 


Financiai/ 

In-kind 


Totai 


$700,000.00 




Worker Online 
Re-skilling 

Office of Partnerships 
for Advanced Skills 
Toronto, ON 


$1/$1.23 


OLT Contribution 


$200,000.00 


Financiai 


Sponsor/Partner 

Contribution 


$245,000.00 


Financiai/ 

In-kind 


Totai 


$445,000.00 




Strategies for 
Employers: Effective 
Strategies for Using 
Technologies in the 
Workplace 
Conferences Board 
of Canada 
Ottawa, ON 


$1/$0.66 


OLT Contribution 


$257,000.00 


Financiai 


Sponsor/Partner 

Contribution 


$170,000.00 


Financiai/ 

In-kind 


Totai 


$427,000.00 




Total 


$1/$1.50 


OLT Contribution 


$1,364,500.00 


Financiai 


Sponsor/Partner 

Contribution 


$2,052,719.00 


Financiai/ 

In-kind 


Totai 


$3,417,219.00 





Note: The data in this tabie was obtained from estimates of project funding made at the time of appiication 
for OLT funding and the figures have not been verified. The estimates represent a smaii sampie of 
OLT-funded projects and these resuits cannot be extrapoiated to the entire popuiation of projects. 



There was considerable variation across the projects, however, as the leveraged amounts 
varied from $0.66 to $3.67 for every dollar spent by the OLT, as compared to the 
expectation that OLT dollars will be matched one-for-one. Also, it is not clear to what 
extent the leveraged funds are incremental. 

Program controls need to be put in place to minimize the risk that funds leveraged 
by OLT projects are not incremental (i.e. would have been used for similar activities 
in the absence of the program), and to safeguard against overlap and duplication. 
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Although the program monitors projects on an ongoing basis throughout the project 
and requires an evaluation to be conducted at the end of the funding period, 
the project sponsor is not required to attest to the incrementality of the project. 

Given that 34% of the non-funded projects proceeded in some form without OLT 
funding, and that these projects obtained alternative funding through other sources 
(e.g. donations, governments), it is important for the OLT to have program safeguards in 
place to prevent, or at least limit, the possibility that the OLT projects would have 
proceeded in the absence of the program. One method of accomplishing this could be to 
require the project sponsors to attest that the project is incremental (i.e., would not have 
gone ahead without OLT funding). 

Similarly, given that some of the unaccepted projects gained funding from other 
sources, it is important for the program to demonstrate that it has safeguards in place 
to prevent, or at least limit, any overlap or duplication of activities funded by other 
sources (e.g. by coordinating activities with NGOs, other levels of government, etc.). 
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6. OLT Contribution to 
Long-Term Development 



This section examines the OLT contribution to the development and/or adoption of 
learning technologies by examining the scope of OLT funded projects, the development 
and use of learning technologies, labour market adaptation, and the introduction of 
learning technologies. 



6.1 Scope of OLT Funded Projects 

Across Canada, a wide variety of organizations received OLT funding. The OLT has 
funded projects throughout Canada. Regionally, the rate of acceptance reflects the rate of 
submission. As shown in Table 6-1, the types of organizations submitting applications 
and receiving funding from the OLT include educational institutions (39%), professional 
groups (28%), and non-profit organizations (24%). 



Table 6-1 

Applications Receiving Funding and Not Receiving Funding by Organization Type 


Organization Type 


Funded 

applicants=283 


Non-funded 

applicants=286 


Public institutions* 


7% 


7% 


Educational institutions** 


39% 


44% 


Professional groups-n 


28% 


13% 


Non-profit organizations-n- 


24% 


26% 


Private companies 


0% 


4% 


Native Groups*** 


2% 


6% 


Total 


100% 


1 00% 



* Public institutions include libraries, municipal/regional government organizations, and public institutions 
** Educational institutions include universities, schools/school boards, and community colleges/technical institutes 
+ Professional groups include professional associations, sector councils and labour union/trade associations 
++ Non-profit includes non-profit/volunteer organizations and community centers 

*** Includes Bands/First Nations Councils/Associations/Organizations and Native Non-profit Organizations 

The diverse range of organizations acting as project sponsors and partners has 
helped to expand the delivery network of the OLT program. The diverse range of 
organizations participating in the OLT has helped to expand the delivery network of the 
OLT because each organizational group has direct or indirect access to different end- 
user groups. Non-profit organizations and educational institutions (universities or 
colleges) commonly acted as project sponsors or project partners, and both 
organizational types have significant access to end-users. 



Summative Evaluation of the Office of Learning Technologies 




OLT projects have reached a wide variety of settings. OLT projects have reached rural 
(13%), urban (33%) and combined urban and rural (53%) settings. Additionally, OLT 
funded projects have promoted learning technologies at local (13%), regional (16%), 
provincial (32%) and national (39%) levels. 

Unaccepted projects that proceeded in the absence of OLT funding projects have a 
narrower scope of reach, and more often operate at a local (28%) or provincial (33%) 
level, rather than regionally (11%) or nationally (17%). 

Also, OLT projects have targeted the community (41%), workplace (27%), home (23%) 
and educational institutions (23%). In general, the setting targeted is directly related 
to the funding initiative objectives. Thus, CLN funded projects primarily targeted 
the community, LTW projects targeted the workplace, and LTI projects targeted the 
workplace, the home and educational institutions. 

Unaccepted projects that proceeded in the absence of OLT funding focused instead on the 
community (50%) and on educational institutions (33%), and were less successful in 
reaching the workplace (17%) and home (5%). 

6.2 Development and Use of Technology 

The OLT has funded projects to promote learning technologies through development, 
testing and use of a wide range of technologies. As shown in Table 6-2, projects across all 
three funding initiatives commonly used the Internet, used/tested/developed a website, 
and developed specialized software or a CD-ROM. Projects also often used computer-based 
training as part of project delivery. The case study evidence and the survey of project 
sponsors also indicated that the OLT has funded projects to promote learning technologies 
through the development, testing and use of a wide range of technologies. 



Table 6-2 

Technology Used, Tested and/or Developed by Projects 




LTW 


LTI 


CLN 


Computer based training 


79% 


86% 


80% 


Internet 


72% 


84% 


96% 


Website 


64% 


89% 


90% 


Speciaiized software 


50% 


58% 


66% 


Multimedia 


50% 


78% 


66% 


CD-ROM 


50% 


61% 


72% 


Intranet 


36% 


55% 


56% 


Digitization 


29% 


60% 


48% 


Video conferencing 


22% 


58% 


46% 
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In comparison to funded projects, non-funded projects were less likely to have 
used/tested/developed: 

• A website (67% non-funded compared to 89% funded); 

• The Internet (56% non-funded compared to 88% funded); or, 

• Specialized software (67% non-funded compared to 75% funded). 



6.3 Labour Market Adaptation 

The OLT program has helped to improve the ability of Canadians to adapt to the needs 
of the labour market primarily through the Learning Technologies in the Workplace 
(LTW) Initiative. The case studies included the following examples of projects assisting 
employees and employers to meet the needs of the labour market: 

• The Alberta Forest Products Association (AFPA)’s Teaching Literacy as a Safety 
Initiative in the Forest Industry project is an LTW project designed to develop a 
technology enhancing training program to increase access to health and safety 
programs for the Alberta forest industry workforce. The project targets forestry 
workers with low literacy skills in remote locations who otherwise would not have 
accessed health and safety programs delivered through conventional classroom 
programs. For this project, AFPA partnered with industry, local colleges and a 
communications company to develop a training CD-ROM for use in the workplace. 
Although the project is too new to be assessed for impacts, project sponsors and 
partners maintained that the overall impacts of the project’s final deliverable will be 
experienced immediately (e.g. new safety, literacy and computer skills for 
employees) and will continue to be realized in the long term. 

• The Office of Partnerships for Advanced Skills (OPAS) Worker Online Re-skilling 
Centre (WORC) is an LTW project to provide Technology Mediated Learning (TML) 
to working adults (especially mid-career, older persons) in need of skills upgrading. 
Although the project was too new to be assessed for impacts, the sponsor and 
partners felt that the use of TML in the culture/art sector (their first targeted sector, 
which includes a large number of small companies) would be effective as a low-cost 
alternative to on-site training, reach rural and remote groups, and provide 
opportunities for end-users to develop broader networks. 

Key informants suggested some additional examples of OLT funded projects aimed at 
adapting Canadians to the job market: 

• Telework Program: A project aimed at using online technology to enhance the 
technological skills and increase employability of groups like single mothers. 

• Rural Capacity Building Through Organic Agriculture: A project that worked to 
increase awareness and use of IT tools and skills by organic farmers in rural areas 
throughout British Columbia. 
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• Industrial Hydraulics E-Learning Program for the Trades: A project to develop a 
fully operational e-leaming system to deliver a hydraulics learning program in the 
workplace and at community access points. 

A high percentage of LTW project sponsors and partners surveyed agreed that the project 
they were involved in helped to: 

• Stimulate the use/application of technology-based learning in the workplace 
(100% sponsor; 100% partner); 

• Develop a more skilled workforce (100% sponsor; 73% partner); 

• Increase employers’ abilities to provide technology-based learning activities 
(90% sponsor; 83% partner). 

LTW funded projects achieved these outcomes by using the workplace as the project 
setting and targeting professional association members (23%), union members (23%), 
businesses (31%), trades (15%) and persons in remote or rural areas (31%). LTW funded 
projects commonly had two main objectives, either the delivery of a training program or 
to increase knowledge or training in a specific field or occupation. 

While LTW had the greatest impact on worker’s labour market adaptation, LTI and CLN 
projects reached the workplace (LTI 39%; CLN 18%). They also targeted the unemployed 
(LTI 8%; CLN 43%), professional association members (LTI 27%; CLN 17%) and union 
members (LTI 12%; CLN 17%). A significant proportion of LTI and CLN projects also 
delivered training programs (LTI 36%; CLN 16%). 

Retraining and skills upgrading appears to be a side effect of some projects funded 
under the Community Learning Networks (CLN) Initiative. The case study evidence 
indicates that both end-users and staff working on the project (for either the project 
sponsors or the partners) developed new technological skills and abilities as a result of 
their participation in the OLT funded project. For example: 

• End-users of the CLN funded project Monashee Learning Community Partnership 
noted that the program had provided local training opportunities that otherwise would 
only have been available to those able to travel 45 minutes to Kelowna. The fact that 
the majority of the courses provided at the center were employment focused was 
appreciated by end-users; and 

• The Fire Suppression and Flagging certification programs resulted in two end-users 
finding employment in these fields. End-users stressed the importance of providing 
programs aimed at re-training the local work force because of the community’s heavy 
reliance on the forest industry. 
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6.4 Introduction of Technology 

The available evidence suggests that OLT projects increased the introduction of 
learning technologies, although it is difficult to determine the extent to which this was 
incremental. To assess whether or not organizations would have been as successful in 
introducing technology to develop skills in the absence of OLT funding, project sponsors 
were asked whether their organization would have undertaken the same project without 
OLT funding. The majority (85%) indicated that they would have been unable to deliver 
the same project scope and activities without OLT funding. About a third (31%) 
indicated that they would have undertaken a similar but modified project, with a smaller 
scope, reduced activities and fewer objectives, without OLT funding. 

As noted in Section 5.4.2, however, it is difficult to determine precisely to what extent 
the projects and their results were incremental (i.e. were in addition to what would have 
occurred without the OLT) because the program does not include safeguards to limit the 
risk that projects and leveraged funds are not incremental, and to safeguard against 
overlap and duplication. 
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7. OLT Contributions to 
Partnership Development 



This section examines the OLT contribution to partnership development including the 
leveraging of resources through partnerships, the sustainability of partnerships, and barriers 
to partnerships. 



7.1 Partnership Development 

The OLT has led to the development of partnerships with private sector, community 
and non-government organizations. The case study analysis indicated the OLT led to 
the development of partnerships through the funding of projects. Sponsors of the case 
study projects also attributed the OLT’s success in developing partnerships to the fact 
that OLT funding emphasized a partnership approach and provided focus and direction to 
community organizations looking to utilize learning technologies. 

The project sponsor survey also supported the conclusions that the OLT has ensured 
collaboration with a wide range of partners by encouraging strong community participation 
and partnerships in the projects funded. Sponsors agreed that the OLT funded project had 
broadened their organizations’ network with other organization/agencies both within (85%) 
and outside (82%) the community (Table 7-1). 



Table 7-1 

Impact of OLT Funded Project on Organization’s Network 


The project... 


Valid N 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Total 

Strongly 

Agree 


Broadened your 
organization’s 
network with other 
organizations/ 
agencies within 
the community 


110 


48% 


37% 


14% 


2% 


0% 


85% 


Broadened your 
organization’s 
network with other 
organizations/ 
agencies outside 
the community. 


106 


41% 


41% 


13% 


4% 


1% 


82% 
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The OLT’s success in promoting partnership development was illustrated by the fact that 
at least one new partnership was developed by 77% of the surveyed project sponsors as a 
result of participation in the OLT program. Moreover, most sponsors had developed 
more than one partnership. On average, 3 new partnerships were developed by each 
partner (as shown in Table 7-2). Project partners came from all sectors including: 

• Private sector organizations (33%); 

• Community-based organizations (75%); and 

• Government organizations (36%). 



Table 7-2 

Average Number of New Partnerships Developed 








LTI 


CLN 


LTW 


Partnerships developed 




Sponsors 


Sponsors 


Sponsors 


with: 


Average 


(N=44) 


(N=41) 


(N=12) 


Community based 
organizations 


3 


3 


3 


1 


Not-for-Profit organizations 


3 


3 


5 


0 


Educational institutions 
(Universities, Colleges, 
Training Providers) 


2 


1 


4 


0 


Private sector organizations 


1 


1 


1 


0 


Industry associations 


1 


1 


1 


1 


Federal government 
organizations 


1 


0 


1 


0 


Provincial government 
organizations 


0 


0 


0 


1 


Other 


0 


1 


0 


0 


Overall Average Number of 
Partnerships 


3 


3 


4 


3 



The key informants also agreed that the OLT has been successful in bringing together the 
private sector, communities, non-governmental organizations and the government to 
partner collaboratively through OLT-funded projects. In part, this success was 
considered to be due to the fact that OLT provides the “big picture” or direction to 
community organizations sponsoring projects. The key informants also confirmed that, 
without access to OLT Pilot funding, many community organizations would not have the 
resources, either staff or financial, to accurately identify community need and to 
subsequently translate that need into meaningful programs. 

Further evidence of the OLT’s emphasis on partnership development was the fact that 
unsuccessful applicants were often refused funding because they did not meet OLT 
objectives and they had not developed enough partnerships for the project to proceed. 
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7.2 Leverage of Resources Through Partnerships 

The evidence from the case studies, surveys of project sponsors and partners, and key 
informant interviews indicated that partnerships for the OLT have leveraged resources in 
a variety of ways, as discussed below. Comments by project sponsors and partners also 
suggest that many of the resources leveraged by OLT partnerships were incremental. It is 
difficult, however, to determine precisely to what extent the leveraged resources were in 
fact incremental (i.e. would not have occurred without the OLT) because of a lack of 
program safeguards in this area (as discussed in Section 5.4.2). 



7.2. 1 Funding and In-kind Resources 

Partnerships for OLT projects have leveraged additional funding and in-kind 
resources for the project. Contributions by partners have included financial (34%) and 
technical support (58%). As discussed in Section 5.4.2, the case study analysis indicated 
that it appears that each dollar of OLT funding leveraged $1.50 from project 
sponsors/partners (see Table 5-3, in Section 5.4.2). 

Also, the case studies and surveys of project sponsors and partners indicated that 
partnerships have increased project recognition and credibility, often leading to additional 
partnerships, funding and in-kind resources for the project. For example, partners of the 
Creative Retirement Manitoba: Senior Learning Network project noted that the profile of 
the project sponsors, the presence of OLT funding, and the partnerships already in place 
had encouraged their participation in the Senior Learning Network. Also, partners in the 
Computer-Based Adult Learning Technologies Project provided a wide variety of 
expertise, with partnering leading to more partnering. 

When the surveyed project sponsors were asked whether or not the funding their organization 
had received from OLT had encouraged funding participation by other funders, the majority 
indicated that OLT funding had encouraged funding from other partners (as shown in 
Graph 7-3). 
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Figure 7-1 

Has the Funding Provided by OLT Encouraged Funding from Other Partners? 




Specific technologies provided to the projects through partnerships include access to the 
Internet (57%), websites (43%), specialized software (43%) and multimedia (33%) 
(as shown in Table 7-4). 



Table 7-3 

Technoiogies Provided by Project Partners 




LTi Project 
Partner 


CLN Project 
Partner 


LTW Project 
Partner 


Total 


Internet 


54% 


58% 


57% 


57% 


Specialized software 


45% 


42% 


43% 


43% 


Computer based training 


54% 


15% 


71% 


43% 


Web site 


27% 


50% 


57% 


43% 


Multimedia 


45% 


25% 


28% 


33% 


CD-ROM 


45% 




43% 


27% 


Specialized devices 


27% 


17% 




17% 


Intranet 


18% 




28% 


13% 


Video conferencing 


18% 


17% 




13% 


Web CT/GIS 


27% 






10% 


Desktop video 
conferencing 


18% 


8% 




10% 


Wireless technology 




17% 


14% 


10% 


Digitization 


9% 


8% 




7% 


Interactive TV 


9% 






3% 
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7.2.2 Content Development 

Partnerships leveraged the content development of OLT projects. When describing the 
main advantages of partnerships, most project sponsors and partners commented that 
partnerships provided multiple benefits to the projects, including the sharing of 
knowledge/skills (44%), and the introduction of more ideas or different perspectives 
(24%) to the content development process. 



Table 7-4 

Main Advantages of Partnerships 




Totai 


Sharing of knowledge/skills 


44% 


Broader target audience 


21% 


More resources 


19% 


Different perspectives 


16% 


increased project effectiveness in reaching target audience 


13% 


More ideas 


8% 


More professionai/student participation 


6% 


More credibiiity 


6% 


More funding 


4% 


Unabie to compiete project without partner 


3% 


increased recognition of project 


3% 



The surveys of project sponsors and partners also indicated general agreement that 

project partnerships: 

• Supported coordination and information sharing (92%); 

• Facilitated content development (78%); and 

• Identified or developed best practices (77%). 

The case study analysis also provided examples of the leveraging of content 

development. 

• The Accessible Adult Learning in Health Professions project, which was developed 
through the Dalhousie University’s Faculty of Health, was able to involve a range of 
internal partners (e.g. five departments within the Health Professions Faculty, plus 
several Centers within the university) and external partners (e.g. Workers’ 
Compensation Board of NS) to advance the faculty’s commitment to providing 
distance health education and increasing program access for those in rural and remote 
communities. 

• The Senior Learning Network Project was undertaken by Creative Retirement 
Manitoba. To obtain senior-relevant content for the Seniors Learning Network 
Internet site. Creative Retirement Manitoba developed multiple informal partnerships 
with organizations throughout Winnipeg that served the senior population. One such 
partnership was with the St. Vital Historical Society. 
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7.2.3 Expanded Delivery Network 

Partnerships for OLT projects have contributed to an expanded project delivery 
network. Partnerships facilitated project delivery through increased access to a broader 
target audience and more targeted end-users, delivery sites, and increased access to 
support staff for project delivery (volunteers). Both formal and informal partnerships 
were considered to have expanded the project delivery network. 

Many projects, particularly CLN projects, utilized informal partnerships to enable the 
project to reach targeted end-users. In these informal partnerships, the partner usually 
provided limited resources to the project. Instead, the informal partner often introduced 
their client group to the services or programs offered through the OLT funded project. 
Monashee Learning and Training Centre: Monashee Learning Community Partnership 
utilized both formal and informal partnerships to enhance the delivery of their services. 
In particular, an informal partnership was developed with the local Community Schools 
Association to develop and distribute a program guide that outlined programs currently 
being offered through both organizations. 

Project accessibility to end-users was also increased through partnerships with 
organizations affiliated with Community Access Program (CAP) sites. This was 
particularly true for the Community Learning Network (CLN) projects, where 44% of the 
project partners were linked to a CAP site (See Table 7-6). For example, the Creative 
Retirement Manitoba: Senior Learning Network partnered with the Middle Church Home 
and Community Connections to obtain access to CAP sites. 



Table 7-5 
Links to CAP sites 




LTi 


CLN 


LTW 


Sponsors 

(N=56) 


Partners 

(N=18) 


Sponsors 

(N=44) 


Partners 

(N=27) 


Sponsors 

(N=14) 


Partners 

(N=13) 


Yes 


5% 


11% 


43% 


44% 


0% 


8% 


No 


95% 


89% 


57% 


56% 


100% 


92% 


Total 


100 % 


100 % 


100 %, 


100 % 


100 %, 


100 %, 



Partnerships also increased access to support staff (volunteers) for project delivery, and this 
supported service delivery in OLT funded projects. Many partnering relationships were 
based on either the project sponsor or the project partner providing trained staff or volunteers 
to provide service delivery to end-users. For example, the Vancouver Community Network 
(VCN): Vancouver Community Learning Network (VLN) partnered with the Technical 
Volunteer Web (TVW). The TVW provided volunteers with expertise in technology who 
were then able to provide instruction on computer use, including accessing the Internet and 
using email, to all other organizations involved in the project. 
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7.3 Partnership Sustainability 

The evidence suggests that many partnerships developed for OLT projects would 
continue in some form after their OLT support end. The case studies and the surveys of 
project sponsors and partners provided considerable evidence that contributions made by 
project partners (e.g. additional funding and in-kind resources, content development and 
expanded delivery networks) were recognized by project sponsors and partners. The surveys 
of project sponsors and partners also described other advantages of their partnerships: 

• Support knowledge/skills sharing (36%); 

• Broaden the target audience (21%); 

• Increase effectiveness in reaching the target audience (13%); and 

• Provide a wider resources base for the project (20%). 

Given the recognition of the value of these partnerships, it seems likely that these groups 
will look for ways to continue partnering in some form after their OLT support ends. 

In addition, 89% of the surveyed project sponsors and partners indicated high levels of 
satisfaction collaborating with project partners in the implementation of the OLT funded 
project. This high level of satisfaction suggests that these groups will look for ways to 
continue partnering. 

The survey of project sponsors indicated that the majority (82%) anticipate 
partnerships developed for the project would extend beyond the completion of the 
project. Looking specifically at LTI funded projects, which generally ended by 1999, 
73% of the partnerships were reported to have extended beyond the life of the project. 

Key informants were also optimistic that the project partnerships created through OLT 
were sustainable. It was noted that OLT funded projects had led to numerous “offshoots” 
or additional projects, and thus partnerships, that are currently in place without OLT 
funding. It was suggested that although OLT funding may not be available to sustain 
partnerships, the momentum, ideals and activities resulting from the project generate a 
commitment that will sustain the partnerships. 

Key informants suggested that OLT could further sustain partnerships by providing 
information to sponsor/partner organizations on how to develop formal partnerships and 
move them forward toward sustainability, as many do not possess the skills and knowledge 
do this. 



7.4 Barriers to Partnerships 

While project sponsors and partners were generally satisfied with the partnerships 
developed for OLT projects, a number of barriers to such partnerships were identified. 
The case study evidence and the surveys of project sponsors and partners indicated that 
project sponsors often lack the required knowledge to develop and mobilize formal 
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partnerships. Lack of knowledge of partnering impacts both the development and 
maintenance of the partnership and can lead to conflicts during project development. 
This is particularly problematic for non-profit organizations and community agencies. 

Supporting the findings that lack of partnering experience impacted partnerships, both 
sponsors and partners commented that disadvantages of partnerships included: 

• Projects having increased organizational and logistical challenges; 

• A significantly slower project development process that required more effort; and 

• Communication breakdowns. 

The case study evidence also indicated that conflicting understanding of project goals 
between partners can significantly impact partnerships and sometimes develop out of 
poorly formali z ed partnerships. Partners interviewed for the case studies noted that these 
conflicts arose, in part, from diverse philosophies and disagreements about organizational 
domains. Conversely, successful partnerships were based on shared philosophies and 
project visions. 

Partner fatigue was also identified as a barrier by the case studies and by surveyed project 
sponsors and partners. Over-commitment and non-delivery of resources by partners have 
impacted the success of partnerships, particularly among non-profit or community 
organizations. Non-delivery of resources appeared to be linked to lack of monies 
available to the partner. When asked to describe the main disadvantages of partnerships, 
sponsors said that a major disadvantage was working with partners who are 
under-resourced or understaffed. 

Delays in receiving funding from OLT (i.e. from the developmental phase to pilot phase 
as a result of the application process) resulted in waning project momentum and interest 
among partners. This sometimes led to partner drop out. Both the case study and key 
informant evidence suggests the need to have a “fast track” procedure for obtaining 
funding between the developmental and pilot phases, for projects that are already 
running. The issue of waning partner interest was more salient when community 
organizations were involved, given the unstable nature of their internal funding and staff 
resources. In particular, the Vancouver Community Network (VCN) and the Office of 
Partnerships for Advanced Skills (OPAS) had difficulty maintaining the partnerships 
developed at the time of the funding application. In the case of the VCN, 
new partnerships needed to be established by the time OLT funding was received because the 
original project partners had either lost their matching funds, closed down or lost resources to 
pay internal staff to champion the project. OPAS found that the delay had affected project 
momentum and the applicability of the results from the developmental phase. Also, OPAS 
had difficulty maintaining partner involvement as partners felt they were losing out on other 
opportunities while waiting to hear if the application was accepted. 

Partner resource levels were identified as another important factor because depleting 
partner resources can lead partners to re-think or forgo project involvement to focus on 
their primary mission. Key informants stressed that the primary barrier to developing and 
maintaining partnerships was funding, particularly partner funding levels. 
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8. Program End-Users 



This section examines the extent to which OLT projects targeted identified equity groups. 

OLT projects have had considerable success in reaching designated equity groups, 
although some groups were not particularly well reached. The case study analysis 
indicated that groups targeted by funded projects include seniors, persons in rural or 
remote communities, persons with disabilities and persons with low literacy skills. 
For example, the Teaching Literacy as a Safety Initiative by the Alberta Forest Products 
Association (AFPA) had a primary objective of increasing literacy skills of workers using 
industry-related knowledge and without alienating the target audience. The self-paced 
CD-ROM utilizes specialized teaching methods and tools to assist low literacy workers 
who would not otherwise participate in classroom based safety training courses. 

The survey of project sponsors also indicated that OLT funded projects had considerable 
success in terms of targeting identified equity groups. For example, 59 percent of project 
sponsors noted that their project had targeted persons in remote/rural areas (as shown in 
Table 8-1). 



Table 8-1 

Equity Groups Targeted by OLT Funded Projects 




LTI 


CLN 


LTW 


All Projects 


Persons in remote/rural areas 


54% 


66% 


54% 


59% 


Women 


43% 


51% 


23% 


44% 


Youth 


15% 


62% 


8% 


32% 


Aboriginal community 


13% 


34% 


8% 


21% 


Seniors 


5% 


36% 


8% 


17% 


Visible minorities 


12% 


28% 


- 


17% 


Persons with disabilities 


12% 


26% 


8% 


17% 



In addition to the defined equity groups, project sponsors/partners also indicated a variety of 
other groups which were targeted as part of the OLT funded project(s). These included: 

• students; 

• unemployed; 

• union members; 

• professional association members; 

• teachers/professors/educators of technology; and 

• literacy workers. 
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It should be noted that while it appears OLT funded projects had good representation in 
terms of remote/rural participants (59% of sponsors noted that their project had targeted 
this group) and women (44%), it was suggested that there were some groups that were 
not particularly well reached: 

• Young mothers: Although several projects noted that they had made specific efforts to 
provide services to both youth and women, it was noted that child-care responsibilities 
often limited the extent to which young mothers could access available 
programs/services. Some project sponsors/partners noted that programs had to retain 
sufficient flexibility to facilitate greater participation for young mothers. Examples 
included location of the project/facility in close proximity to daycare, and/or providing 
“quiet space” for mothers for nursing and/or other child related activities. 

• ESL populations: Some project sponsors/partners noted difficulty in attracting ESL 
populations/clients to OET projects. Eack of familiarity with technology and 
language issues were cited as key barriers to participation. ESE populations were 
also noted as requiring both language and computer training in order to obtain the 
maximum value from the provided programs. 

• Aboriginal clients: In the case of the Aboriginal clients, the key missing resources 
needed to facilitate access to learning technology were said to be education, teacher 
training and access to financial and technological resources. To overcome these 
issues, informants suggested a more directed or “targeted” effort by OET to meet with 
these groups and create links between the community access programs that serve 
them and OET programs. 

Non-funded projects also targeted a number of the identified equity groups including: 

• Persons in remote areas (55%); 

• Youth (44%); 

• Aboriginal community (44%); and 

• Women (39%). 

The evidence indicates that OLT projects have worked to identify and respond to the 
needs of the end-user groups they target and have been innovative in adapting learning 
technology to be relevant and meaningful to the users. The case studies noted a variety 
of ways in which projects have worked to identify and respond to the needs of the 
end-user groups targeted by the project: 

• Eor the Vancouver Community Learning Network (VEN) project. The Vancouver 
Community Network (VCN) adapted Internet portals and computer training to diverse 
groups of end-users including seniors, cultural workers, and multi-barrier inner city 
residents. In each instance the content and format of the Internet portals was directed by 
the partner organization’s mandate and need. Additionally, the VEN coordinated 
workshops and guest speakers to illustrate the applicability and use of learning 
technologies to the end-users and facilitate participant driven content of the Internet 
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portals. By making technology relevant to the concerns, interests and needs of the 
community, VCN decreased the barriers faced by individuals accessing the technology. 

Project partners indicated that the project had helped end-users by: 

- Decreasing social isolation (e.g. by increasing connections with family 
and friends); 

- Facilitating lifelong learning (e.g. by increasing the confidence and desire to 
learn); and 

- Helping individuals better adapt to the labour market (e.g. by teaching 
transferable skills and assisting in job search and networking). 

The end-user focus group conducted for this project with seniors confirmed that the 
project had decreased seniors’ social isolation by teaching them how to e-mail friends 
and family, increased social interaction within the Carnegie Community Centre 
(where a computer room had been developed by the project and staffed by VCN 
volunteers to allow public drop-in access and allow drop-ins to obtain computer 
training and one-on-one assistance) and increased their interest in capacity to use 
learning technology. 

• For the Computer-Ba^ed Adult Learning Technologies Project, the Fanshawe College 
developed a learning model that applied a computer mediated learning (CML) 
process to adult learners facing various barriers or limited access to learning 
technologies. The end-user groups included the unemployed/underemployed, single 
parents, learners with disabilities, low-literacy learners, and individuals located in 
remote rural areas. A non-credit course was offered over the Internet to adult 
learners, many of whom were members of equity groups, to develop skills in the 
Internet, conferencing software and an audio-tactile network (ATN). 

Although an end-user focus group could not be organized for this case study, due to a 
lack of contact information for the majority of individuals who completed the course, 
the case study and the project’s final evaluation report indicated that project end-users 
benefited from participation through increased language, decision-making, research 
and problem-solving skills. Also, the program helped to develop self-esteem and 
clarify career paths for end-users. One student had successfully acquired certification 
through the program, while two others went on to become facilitators for other CML 
programs at one of the community agencies. 

The surveyed project sponsors and partners also confirmed that their OLT projects had 
developed new ways of using learning technology to reach target audiences by: 

• Facilitating content development (78%); 

• Testing the effectiveness of learning technologies in various settings (83%); 

• Developing, testing and/or implementing new learning models (82%); and 

• Conducting research on issues related to learning technologies (70%). 
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The key informants agreed that projects funded through the OLT initiatives of CLN, 
LTI and LTW have had a positive impact on a diverse group of end users. Within the 
community, the CLN initiative was said to assist in both identifying and meeting 
community needs through pilot and developmental project funding. One of the OLT 
initiative’s greatest strength was said to be that it facilitated programs that “brought 
together people who were experiencing the same challenges”, especially rural groups and 
people in job transition. Further, through OLT funded projects, techniques have been 
developed and used to bring new technology to marginalized groups who would not 
normally use learning technology. Key informants commented that access to new 
technology has helped such adult learners as the under/unemployed, seasonal workers, 
individuals with low literacy/numeracy skills, low income groups. First Nations groups, 
seniors, single mothers, and people from rural communities and resource-based 
communities. Youth have also been reached at the community level by OLT funded 
initiatives. Examples given by key informants of OLT funded projects that reached 
targeted audiences included the Family Literacy Project and the distance e-learning 
program in science for Aboriginal students. 
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9. OLT Program Impact 



This section examines the impacts of the OLT program by examining project 
sustainability and the impact of the program on project sponsors and partners, end-users 
and the community. Once again it should be noted, however, that it is difficult to 
determine precisely to what extent the impacts would not have occurred without the OLT 
because of the lack of program safeguards to limit the risk that projects and leveraged 
funds were not incremental (as discussed in Section 5.4.2). 



9.1 Project Sustainability 

The evidence suggests that many OLT-funded projects would continue in some form 
after OLT support ends, or produced outputs/products that will be used after OLT 
support ends. An assessment of project sustainability for the ten case study projects 
indicates that: 

• Two projects either have continued or plan to continue; 

• Five projects involved one-time developmental activities (e.g. a pilot test, testing a 
new learning model, designing a technology-enhanced health and safety program) 
had either just terminated or plan to terminate when the final deliverable is complete, 
although the resulting products are either in use, or expected to be used after project 
completion; and 

• Three projects were too new to determine whether they will be sustainable. 

The surveyed project sponsors were asked to indicate the extent to which their current project 
would continue once OLT funding expired. As highlighted in Chart 9-1, more than 
three-quarters (78%) of project sponsors felt that their project would continue in some form 
after their OLT support ends. 

Chart 9-1 also indicates that a significant proportion of the surveyed project sponsors 
(54%) noted that they expected the proportion of their project that would continue would 
constitute more than fifty percent of the activity funded by OLT. About a third (35%) 
indicated that fifty to ninety percent of their project would continue and nineteen percent 
(19%) indicated that one hundred percent or the entire project was expected to continue. 

Many of the key informants felt that it was too early in the life of the projects to know 
whether they would be sustainable. However, some believed that it was already evident 
that some projects are sustainable in the long term, while others are not. Key informants 
also noted numerous “offshoots” have led to additional projects that are currently in place 
without OLT funding. 



Summative Evaluation of the Office of Learning Technologies 




For many informants the biggest issue surrounding sustainability was funding. Necessary for 
sustainability is the need for project sponsors to assume some of the project cost currently 
funded through OLT, or continued involvement by OLT or government, as there are 
insufficient resources at the local level to sustain projects. 

Key informants suggested that the OLT could further sustain projects by funding projects 
for a longer durations and incorporating the notion of sustainability as a condition of 
funding, as a way of addressing the notion held by some project sponsors that OLT funds 
“pilots” and that completing the pilot means that the project is over. 



Figure 9-1 

Proportion of the Original Project Expected to Continue 
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9.2 Impact on Project Sponsors and Partners 

Most project sponsors indicated that their project would not have gone ahead without 
OLT funding or would have proceeded on a more limited scale. As noted in Section 
6.4, the majority (85%) of project sponsors surveyed noted that they would not have 
undertaken the same project without OLT funding. For example: 

• Some sponsors noted that they would have had fewer partners had they not received 
OLT funds; 

• Some partners indicated that their project would have been of shorter duration had 
they not received OLT funding; and 

• Several project sponsors noted that the scope of their project would have been 
considerably reduced had they not received OLT funds. 

Several project sponsors noted that the provision of OLT funds helped ensure that 
adequate evaluations were completed during the project to identify lessons learned, 
as well as to provide direction for required changes in program delivery. 
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Project sponsors indicated that interaction with the OLT had increased their 
knowledge of learning technologies. In addition to the impact of the funding, project 
sponsors were asked to comment on the impact of working with the OLT for their own 
organization. Project sponsors were positive in terms of the information acquired 
through their interaction with OLT. For example, interaction with OLT increased 
sponsors knowledge of: 

• New approaches in learning technologies (88%); 

• Learning technologies information (80%); and 

• Trends in the use of learning technologies (59%). 



9.3 Impact on End-Users 

There is a need for better outcome tracking at the end-user level. The evaluation of 
program impacts on end-users was limited because the program never required the 
collection of end-user client data from the funded projects and end-user/client 
information was unavailable from OLT administrative databases. The requirement for 
end-user data collection will be included in the new RMAF. It will be important, 
however, to test the new tracking methods in a variety of end-user settings to ensure that 
the new methods can be easily applied without undue impact on project administration or 
on program reach. 

The available evidence indicates that OLT projects had beneficial impacts for end-users. 
In order to partially compensate for the unavailability of end-user data, focus groups were 
conducted with three of the four CLN case studies. Each focus group consisted of nine 
end-users and identified the following positive impacts on end-users: 

• Vancouver Community Network: Project targeted on individuals with low income 
and literacy skills. The focus group was conducted with seniors. The major impact 
on senior end-users was decreased social isolation through learning Internet and 
e-mail skills, increases direct social interaction at a drop-in computer center 
developed and staffed through the project, and increased interest and capacity to use 
learning technologies; 

• Monashee Learning and Training Centre: Project targets on increasing access 
to lifelong learning and to increase the presence of on-line businesses. Benefits to 
end-users were increased confidence in learning and a reduced aversion to using 
computers. Some project participants are teaching at the center; and 

• Creative Retirement Manitoba: Project aims to provide seniors with necessary 

government and community information by on-line access, combined with courses 
and mentoring programs to improve seniors’ computer skills. Benefits to seniors 
include reduced social isolation, intellectual stimulation and a feeling of 
independence and accomplishment. 
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As indicated in Table 9-2, the surveyed project sponsored and partners indicated that 
OLT funding had achieved the following: 

• Helped workers take advantage of technology (82%); 

• Facilitated lifelong learning in the community (81%); and 

• Helped develop a more skilled workforce (77%). 



Table 9-2 

Project Sponsor/Partner Perceptions of Project 
Impacts on End-Users on Selected Issues* 




Valid N 


% Disagree/ 
Strongly 
Disagree 


% 

Neutral 


% Agree/ 
Strongly 
Agree 


Helped workers take advantage of 
technology. 


150 


3% 


15% 


82% 


Facilitated lifelong learning In the 
community. 


145 


3% 


16% 


81% 


Helped develop a more skilled 
workforce. 


135 


4% 


19% 


77% 


Enabled workers to participate more 
fully In the community. 


122 


9% 


22% 


69% 


Addressed regional employment/sklll 
needs. 


126 


10% 


22% 


68% 


Contributed to the creation of 
employment opportunities In the 
community. 


118 


13% 


33% 


54% 



* Excludes don’t know/no opinion responses 



Site visits conducted for the case studies also included interviews with the project 
sponsors and partners, and these interviews cited further examples as to how OLT funded 
projects benefited end-users. As noted in Section 6.3: 

• The Alberta Forest Products Association (AFPA)’s Teaching Literacy as a Safety 
Initiative in the Forest Industry project is developing a CD-ROM to provide health and 
safety training to forestry workers for low literacy skills in remote loeations. When the 
deliverable is completed, the project sponsors and partners maintain that the overall 
impacts will include new safety, literacy and computer s ki lls for these employees. 

• The Office of Partnerships for Advanced Skills (OPAS) Worker Online Re-skilling 
Centre (WORC) is providing Technology Mediated Learning (TML) to working 
adults (especially mid-career, older persons) in need of skills upgrading. The sponsor 
and partners felt that the use of TML in the culture/art sector (their first targeted 
sector) will be effective as a low-cost alternative to on-site training, reach rural and 
remote groups, and provide opportunities for end-users to develop broader networks. 
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9.4 Impact on the Community 

The available evidence indicates that OLT projects have benejicial impacts on the 
community. The surveyed project sponsors and partners also cited community-level 
benefits associated with OLT funded projects. Project sponsors and partners agreed that 
funded projects have: 

• Broadened access to learning opportunities (95%); 

• Supported community capacity building (82%); 

• Raised awareness of learning opportunities (78%); 

• Facilitated the development of community expertise in learning technologies (74%); 
and 

• Helped develop the communities’ technological infrastructure (66%). 

The OLT program contributed to improved accessibility to learning technologies. 
Almost half (44%) of all CLN project were directly linked, through the project partner 
or sponsor, to a CAP site. Additionally, by funding CLN projects facilitated by 
community colleges or institutes (22%) and not-for-profit organizations (38%) that 
serve the community, OLT has further increased accessibility to learning technologies. 
With respect to the many objectives of CLN funded projects, most were aimed at 
increasing learning skills (23%), reducing barriers (23%) or increasing accessibility to 
technologies (27%). 

The case study evidence provides some specific examples of OLT funded projects that 
increased accessibility to learning technologies by providing learning opportunities 
through the use of various technologies. 

• Accessible Adult Learning in the Health Professions Program (Dalhousie University): 
Through the Faculty of Health Professions, the project provided distance education 
courses to students at the undergraduate and graduate level using teleconferencing, email, 
and Internet or web-based learning. 

• Computer-Based Adult Learning Technologies (Fanshawe College): The project 
utilized a learning assessment and recognition (PLAR) strategy and various 
technologies (e.g. Internet, conferencing software and audio-tactile network) to assist 
equity groups to learn new technical skills. End-user groups included: 

- unemployed and underemployed individuals; 

- women and single parents; 

- learners with disabilities; 

- individuals with low levels of literacy and English proficiency; and 

- individuals in remote or rural areas. 
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10. Conclusions and Areas 
for Improvement 



This section presents a summary of the main findings of the evaluation and the areas 
identified for improvement. 



10.1 Summary of Main Findings 

There continues to be a need for a national program that encourages the use of learning 
technologies to develop new technical skills and upgrade old skills - particularly in smaller, 
isolated communities. Currently, the OLT appears to be almost unique in Canada in funding 
projects that link learning technologies with non-institutional learning, such as community or 
lifelong learning, while maintaining an overall program focus on skills development and 
training for innovation. 

Stakeholders also see a need for OLT funding and believe that program objectives are 
still valid. In the future, the federal government and the OLT are seen to have an 
important role in: 

• Facilitating infrastructure development and access; 

• Developing and facilitating partnerships; and 

• Providing financial assistance and funding. 

A wide variety of organizations applied for and received OLT funding. The types of 
organizations submitting applications and receiving funding from the OLT include 
educational institutions (39%), professional groups (28%), non-profit organizations 
(24%), public institutions (7%) and Native groups (2%). 

The OLT helped develop partnerships with private sector, community and non-government 
organizations. The case study and survey analyses indicate that OLT funding supported 
partnership development, for example by making it possible for the project to be 
developed or to be developed with a broader scope and impact. At least one new 
partnership was developed by 77% of the surveyed project sponsors as a result of 
participating in the OLT program, and most sponsors developed more than 
one partnership. The range of project partners included community-based organizations 
(75%) and the private sector (33%). 

Project sponsors attributed the OLT’s success in developing partnerships to the emphasis 
the program places on the partnership approach and the ability of the program to provide 
focus and direction to community organizations looking to utilize learning technologies. 
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Project partners made a variety of contributions to OLT projects, including additional 
funding, in-kind-resources, content development, and expanded delivery networks. 
The case study analysis suggests that each dollar of OLT funding leveraged an average of 
$1.50 from project sponsors/partners. 

OLT funded projects have contributed to increased access to and use of learning 
technologies among a wide variety of designated equity groups. The OLT has funded 
projects to promote learning technologies through the development, testing and use of 
a wide range of technologies (including computer-based training, the Internet, 
the development/use/testing of a website, and the development of specialized software 
or CD-ROM). For example, the many objectives of CLN funded projects included 
reducing barriers (23%), increasing learning skills (23%) and increasing accessibility 
to technologies (27%). 

The evidence indicates that OLT projects had considerable success in reaching 
designated equity groups (e.g. persons in remote/rural areas, women, youth, persons with 
low literacy skills, seniors, and persons with disabilities). The evidence also indicates 
that OLT projects have worked to identify and respond to the needs of end-user groups 
identified by the project. 

The available evidence on end-user impacts indicates that OLT projects benefited 
end-users. End-user focus groups for three of the four CLN project case studies 
identified a number of impacts, including increased confidence in learning and using 
computers, and reduced social isolation. The survey of project sponsors indicated that 
OLT projects have increased access to learning technologies and have helped workers 
take advantage of technology (82%), facilitated lifelong learning in the community 
(81%), and helped develop a more skilled workforce (77%). 

OLT funding appears to be an important factor in the development of projects aimed at 
using technology to promote learning. Only 6 % of the non-funded projects proceeded in 
full without OLT funding, and another 28% proceeded in a reduced form. The rest (66%) did 
not proceed. The major reasons given by non-funded applicants for projects not proceeding 
after not being accepted by the OLT was the lack of alternative sources of funding. 

The evidence suggests that the partnerships and projects developed through the OLT 
would continue in some form after their OLT funding support ends. There is 
considerable evidence that contributions made by project partners (e.g. additional 
funding, in-kind resources, content development, expanded delivery networks) were 
recognized by project sponsors and partners. Also, project sponsors and partners 
indicated high levels of satisfaction with their experiences collaborating for the OLT 
funded project. The value and satisfaction associated with their partnering experience 
under the OLT suggests that these groups will look for ways to continue partnering in 
some form after their OLT funding support ends. 

In support of this conclusion, the survey of project sponsors indicated that the majority 
(82%) anticipated that partnerships developed for the OLT project would extend beyond 
the completion of the project. The experience with LTI funded projects, which generally 
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ended in 1999, also corroborates this conclusion (with 73% of the partnerships reported 
to have extended beyond the life of the project). 

The evidence also suggests that many OLT projects would continue in some form or 
produced outputs/products that will be used after OLT support ends. For example, 
evidence from the ten case studies indicates that two projects either continued or planned 
to continue, five projects involved one-time development activities to produce 
outputs/products that are either in use or expected to be used after project completion, 
and the remaining three projects are too new to determine whether they will be 
sustainable. Similarly, more than three-quarters (78%) of the surveyed project sponsors 
felt that their project would continue in some form after the end of OLT funding. 



10.2 Areas Identified for Improvement 

The broad over-reaching objective of the OLT needs to be recast in more realistic terms. 
The OLT was established with an annual budget of $6 mi llion. The current objective of the 
OLT is broad and over-reaching in aiming to reduce the economic and social divide between 
those who have computer skills and those who do not. The current over-reaching objective 
needs to be recast to be something that a program with a limited budget can realistically 
achieve. An example might be to make the objective to improve computer skill sets of 
end-users and reduce their reluctance/inability to use newer technologies. 

Program controls need to be put in place to minimize the risk that funds leveraged 
by OLT projects are not incremental, and to safeguard against overlap and 
duplication. Although there is evidence that OLT projects are leveraging additional 
funds, the current program design does not include safeguards to reduce the risk that 
leveraged funds are not incremental (i.e., would have been used for similar activities 
in the absence of the program). 

There is a need for the program to be able to demonstrate that it has safeguards in place to 
prevent, or at least minimize, the risks that leveraged funds are not incremental and that 
OLT projects would have proceeded in the absence of the program. One method of 
accomplishing this could be to require the project sponsor to attest that the project is 
incremental (i.e. would not have gone ahead without OLT funding). 

Similarly, given that some of the unaccepted projects gained funding from other sources, 
there is a need for the program to be able to demonstrate that it has safeguards in place to 
prevent, or at least limit, any overlap or duplication of activities funded by other sources 
(e.g. by coordinating activities with NGOs, other levels of government, etc.). 

While OLT has been able to reach designated equity groups, certain groups may 
require additional program effort to be adequately reached. While OLT funded 
programs have been relatively successful in reaching persons in remote/rural areas, 
women and seniors, there has been less success in reaching the Aboriginal community, 
young mothers and ESL populations. Aboriginal groups, especially, may require a 
targeted initiative to address the multiple issues identified in serving this population. 
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There is a need for better outcome tracking at the end-user level. There is a need to 
provide for the tracking of end-users, for example by requiring project participants 
to register with project staff. The requirement for end-user data collection will be 
included in the new RMAF. It will be important, however, to test new tracking methods 
in a variety of end-use settings to ensure that the new methods can be easily applied 
without undue impact on project administration by sponsors or on program reach. 

Project sponsors and partners lack experience partnering. Partnership activities 
required considerable resources and were particularly challenging for organizations that 
were inexperienced in partnering or that lacked the required knowledge to develop and 
mobili z e formal partnerships (i.e. non-profit and community organizations). Lack of 
knowledge and partnering experience impacts both the development and maintenance 
of partnerships and can lead to conflicts during project development. In general, 
organizations would like more direction and assistance from the OLT in terms of the 
development and maintenance of partnerships. 

OLT staffing levels and staff turnover have impacted the program. There is a perception 
that there is insufficient OLT staff to manage current projects. Further, high staff 
turnover within the OLT has created an impression that staff are not sufficiently familiar 
with the program or with learning technologies. Generally, project sponsors would like 
more interaction and direction from OLT staff. 

There is a strong perception that OLT should allocate more resources to understanding 
and disseminating best practices and lessons learned. It is felt that OLT can act as a 
data warehouse for learning technology information, research and findings from projects. 
Stakeholders also requested additional opportunities for project sponsor interaction and 
information sharing between and within project initiatives. 

Delays in receiving OLT funding (i.e., from the developmental phase to pilot phase as a 
result of the application process) resulted in waning project momentum and interest 
among partners, sometimes resulting in partner drop out. Informants and sponsors 
suggested that there be a “fast track” procedure for obtaining funding between the 
developmental and pilot phases, for projects that are already running. The issue of 
waning partner interest was more salient when community organizations were involved, 
given the unstable nature of their internal funding and staff resources. 
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Appendix A 
Evaluation Matrix 



Evaluation Questions 


Indicators 


Methodologies 


1. Are the objectives and 
mandate of the OLT still 
relevant given the current 
situation? Is it realistic to 
aim at closing the Digital 
Divide? What should be the 
role of the federal 
government in the future? 


• Opinion of experts in the learning 
technology field 

• Analysis of needs of target audience 

• Assessment of learning technologies 
status, gaps and trends 

• Availability of technical 
infrastructure (computers, networks, 
internet connection, technical 
support systems, CAP sites) 


• Document Review 

• Interviews 

• Survey of sponsors 

• Survey of unfunded projects 

• Survey of partners and 
community leaders 

• Panel of Experts 


2. How and to what extent 
does OLT contribute to the 
development and use of 
learning technologies 
in Canada? 

Does the program improve 
the ability of Canadians to 
adapt to the needs of the 
labour market? 


• Observations of sponsors 
and partners 

• Observations of end-users 

• Distribution of LTI and CLN 
funded projects 

funding levels 
settings (home workplace, 
community learning centres) 
end-users 

regional differences 

• Summary of policy forms held 

• Opinion of experts in the learning 
technology field 

• Regional perspective 

Socio-economic well-being 
Employment Shortages 
and needs 
Skills requirements 
Resources, capacities and 
infrastructure 

• Learning technologies developed 
and used without OLT funding 

settings (home workplace, 
community learning centres) 
end-users 

regional perspective 


• Document Review and 
Administrative Data Analysis 

• Interviews 

• Survey of sponsors 

• Case studies 
(with focus groups) 

• Survey of unfunded projects 

• Survey of partners 
and community 


3. To what extent has the OLT 
succeeded in the 
development of partnerships 
with private sectors, 
communities, 
non-governmental 
organizations (NGO) and 
governments? Have the 
partners been contributing to 
expand delivery networks 
and content development? 
What are the principal 
determinants of success and 
barriers to partnership? 


• Observations of sponsors 
and partners 

• Role of volunteers/staff and other 
community resources 

• Number of partners involved 
and roles 

• Resources contribution to project 

• Advantages and disadvantages 
of partnership 

• Broadened networks 

• Number of new networks 

• Community-based 
organization involvement 

• Evidence of synergy between 
OLT projects in the community 
of elsewhere 


• Document Review and 
Administrative Data Analysis 

• Interviews 

• Survey of funded projects 

• Case Studies 

• Survey of unfunded projects 

• Survey of partners 
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Evaluation Questions 


Indicators 


Methodologies 


4. Who and where are the adult 
learners reached hy OLT 
projects? Are they part of 
Equity Groups? Have their 
needs been met? What type 
of activities/services/tools/ 
practices have been 
provided to meet 
their needs? 


• Observations of sponsors 
and partners 

• Distribution of LTI and CLN 
funded projects 

funding levels 
settings (home workplace, 
community learning centres) 
end-users 

regional differences 
involvement of end-users 


• Document Review and 
Administrative Data Analysis 

• Interviews 

• Survey of funded projects 

• Case studies 
(with focus group) 

• Survey of partners and 
community leaders 


5. Have the funded projects 
demonstrated the 
applicability and 
effectiveness of innovative 
use of learning technologies 
and learning models? 
Specifically: 

In various settings such as 
the workplace, the 
community learning centres 
and homes. 

For individuals with special 
learning needs or individuals 
facing barriers to their 
learning (low-income, single 
parents, unemployed, 
part-time and 
school dropouts). 

To what extent have OLT 
projects been replicated 
elsewhere as models? 


• Observations of sponsors 
and partners 

• Distribution of LTI and CLN 
funded projects 

funding levels 
settings (home workplace, 
community learning centres) 
end-users 

regional differences 

• Increase knowledge of 
LT opportunities 

• Partner assessments of effectiveness 


• Document Review and 
Administrative Data Analysis 

• Interviews 

• Survey of funded projects 

• Case studies 
(with focus group) 

• Survey of unfunded projects 

• Survey of partners and 
community leaders 

• Panel of Experts 


6. Are the projects and 

partnerships created through 
the OLT sustainable? 

Would the organization have 
been able to make the 
progress that they have in 
introducing technology to 
develop skills if they had not 
had access to OLT funding? 


• Observations of sponsors 
and partners 

• Relative importance of OLT funding 

New resources committed 

• Technical expertise developed 

• Project supported through a CAP 
site (CLN) 

• Projects beyond the expiry of 
OLT funding 

Duration 

Proportion of operations 
that continue 
Level of co-operation 
between partners 
Frequency of meetings 


• Administrative Data Analysis 

• Interviews 

• Survey of funded projects 

• Case studies 

• Survey of unfunded projects 

• Survey of partners 


7. What did OLT do to 
promote learning 
technologies? Did it 
increase the sharing of 
knowledge and awareness 
about effective use of 
learning technologies and 
new approaches? 


• Observations of sponsors 
and partners 

• Distribution of OLT awareness 
funded projects 

Funding levels 

Settings (workshops, seminars, 
forums, promotion products, 
web-bases, LT tools) 

Target audience 
Regional differences 


• Document Review and 
Administrative Data Analysis 

• Interviews 

• Survey of funded projects 

• Case studies 

• Survey of unfunded projects 

• Survey of partners and 
community leaders 
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Evaluation Questions 


Indicators 


Methodologies 


8. To what extent did OLT 
contribute to an 
improvement of 
accessibility of learning 
technologies? By end-user 
groups, equity groups and 
regions? Is there new and 
enhanced capacity of 
partners and networks to 
serve clients using LTs? 


• Observations of sponsors 
and partners 

• Distribution of LTI and CLN 
funded projects 

funding levels 
settings (home workplace, 
community learning centres) 
end-users 

regional differences 

• Comparative analysis of CAP sites 
and community colleges who 
received and did not receive 
OLT funding 


• Document Review and 
Administrative Data Analysis 

• Interviews 

• Survey of funded projects 

• Case studies 

• Survey of unfunded projects 

• Survey of partners and 
community leaders 


9. What are the costs 

associated with each type of 
OLT funded projects and 
how does it compare with 
non-OLT projects? Is the 
OLT management model 
cost-effective? 


• Observations of sponsors 
and partners 

• Opinion of experts in the learning 
technology field 

• Comparative analysis of CAP sites 
and community colleges who 
received and did not receive 
OLT funding 


• Document Review and 
Administrative Data Analysis 

• Interviews 

• Survey of funded projects 

• Case studies 

• Survey of unfunded projects 

• Survey of partners and 
community leaders 
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Appendix B 
Summary of Case Studies 



This appendix provides an overview of the key issues/findings drawn form the ten OLT 
projects sites visited as part of the evaluation. It should be noted that due to differences 
in the sites (i.e. some sites have been operational for 1-2 years, while other sites had been 
operational for only a few months, or the project had ended) it was not possible to 
identify outcomes/barriers for all projects. 

The following issues are presented: 

> Project overview; 

> Target population/client group; 

> Client needs (objectives of the project); 

> Extent to which client needs/project objectives were met; 

> Identification of barriers/factors that affect achievement of the objectives; and, 

> Other issues. 
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